Oct 30 19:03:40.356986 ip-10-0-135-68 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:03:40.725088 ip-10-0-135-68 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:40.725088 ip-10-0-135-68 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:03:40.725088 ip-10-0-135-68 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:40.725088 ip-10-0-135-68 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:40.729199 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729119 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.729199 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729200 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729206 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729209 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729213 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729215 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729218 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729221 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729224 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729226 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729229 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729232 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729235 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729237 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729240 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729244 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729249 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729252 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729256 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729259 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729261 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.729299 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729264 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729266 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729269 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729272 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729275 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729282 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729285 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729288 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729291 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729293 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729296 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729298 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729302 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729305 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729308 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729311 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729314 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729316 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729319 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.729791 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729322 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729324 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729327 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729329 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729332 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729335 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729337 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729340 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729342 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729345 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729347 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729350 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729353 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729355 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729358 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729364 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729367 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729369 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.730231 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729372 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729375 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729378 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729380 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729383 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729385 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729388 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729390 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729393 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729396 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729399 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729401 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729404 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729406 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729409 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729411 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729414 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729417 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729419 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729421 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.730784 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729424 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729426 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729429 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729432 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729435 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729438 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729441 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729445 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729449 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729452 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729455 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729457 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729460 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729463 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729465 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729468 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729470 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729472 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729475 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.731535 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729477 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729480 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729483 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729486 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729489 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729491 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729494 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.729496 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730283 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730290 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730293 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730296 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730298 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730301 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730304 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730307 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730309 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730313 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730316 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730324 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.732018 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730328 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730331 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730333 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730336 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730339 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730342 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730344 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730347 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730350 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730352 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730355 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730357 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730360 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730362 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730365 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730368 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730370 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730373 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730375 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730377 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.732494 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730380 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730383 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730386 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730388 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730391 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730393 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730396 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730399 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730401 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730404 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730407 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730410 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730413 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730416 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730419 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730421 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730424 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730426 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730429 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.732991 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730431 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730434 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730436 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730439 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730441 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730444 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730446 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730450 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730454 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730456 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730459 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730461 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730464 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730467 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730469 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730472 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730474 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730477 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730480 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.733479 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730482 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730485 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730487 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730490 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730493 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730496 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730499 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730504 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730507 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730509 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730512 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730514 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730517 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730519 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730521 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730524 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730527 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730529 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730532 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730534 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.733942 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730537 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730539 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730543 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730548 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730551 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730555 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730559 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730562 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730564 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730567 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730570 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730572 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730575 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730577 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730580 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.730582 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731155 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731162 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731172 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731181 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.734416 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731185 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731188 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731191 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731194 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731196 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731199 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731202 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731204 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731207 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731209 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731212 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731215 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731217 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731220 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731222 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731225 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731227 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731230 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731233 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731235 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.734910 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731238 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731240 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731243 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731245 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731248 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731250 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731253 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731256 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731258 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731261 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731265 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731267 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731276 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731279 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731281 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731284 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731286 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731289 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731291 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.735377 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731294 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731296 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731299 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731301 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731304 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731306 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731309 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731312 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731314 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731317 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731319 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731322 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731324 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731327 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731329 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731331 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731334 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731336 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731339 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731342 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.735840 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731344 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731347 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731349 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731352 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731355 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731357 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731365 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731368 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731371 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731374 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731376 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731379 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731381 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731384 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731388 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731392 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731395 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731397 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731400 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.736307 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731402 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731405 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731407 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731410 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731412 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731415 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731417 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731420 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731423 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731425 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731428 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731431 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731436 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731439 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731441 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731444 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731447 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731450 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731452 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.736869 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731455 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731464 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731467 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731469 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.731472 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731561 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731568 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731585 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731589 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731594 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731597 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731601 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731605 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731609 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731612 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731615 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731618 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731621 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731625 2582 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731628 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731631 2582 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731634 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731637 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:03:40.737326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731644 2582 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731647 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731652 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731655 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731661 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731665 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731668 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731672 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731687 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731691 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731694 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731703 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731706 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731710 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731714 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731717 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731720 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731723 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731726 2582 flags.go:64] FLAG: --enable-server="true" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731729 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731736 2582 flags.go:64] FLAG: --event-burst="100" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731739 2582 flags.go:64] FLAG: --event-qps="50" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731742 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731745 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:03:40.737891 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731748 2582 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731752 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731756 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731759 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731762 2582 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731765 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731768 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731771 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731775 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731777 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731782 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731785 2582 flags.go:64] FLAG: --feature-gates="" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731790 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731794 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731797 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731800 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731803 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731806 2582 flags.go:64] FLAG: --help="false" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731809 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-68.ec2.internal" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731812 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731821 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731825 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731828 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:03:40.738512 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731832 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731835 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731838 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731841 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731843 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731847 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731850 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731853 2582 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731856 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731859 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731862 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731865 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731867 2582 flags.go:64] FLAG: --lock-file="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731870 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731873 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731876 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731882 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731885 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731888 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731892 2582 flags.go:64] FLAG: --logging-format="text" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731895 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731900 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731903 2582 flags.go:64] FLAG: --manifest-url="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731906 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731910 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:03:40.739072 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731913 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731917 2582 flags.go:64] FLAG: --max-pods="110" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731920 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731923 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731926 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731935 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731938 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731942 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731944 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731951 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731955 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731958 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731961 2582 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731964 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731967 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731970 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731973 2582 flags.go:64] FLAG: --port="10250" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731976 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731979 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040809bb5903bb00a" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731982 2582 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731985 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731988 2582 flags.go:64] FLAG: --register-node="true" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731991 2582 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731994 2582 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.731997 2582 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732000 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:03:40.739699 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732004 2582 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732008 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732013 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732016 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732019 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732022 2582 flags.go:64] FLAG: --runonce="false" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732025 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732028 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732032 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732035 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732038 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732041 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732051 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732055 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732058 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732060 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732063 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732066 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732069 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732072 2582 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732075 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732080 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732084 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732086 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732093 2582 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:03:40.740309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732096 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732099 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732101 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732104 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732107 2582 flags.go:64] FLAG: --v="2" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732112 2582 flags.go:64] FLAG: --version="false" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732116 2582 flags.go:64] FLAG: --vmodule="" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732122 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732125 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732325 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732330 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732333 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732336 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732340 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732343 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732345 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732348 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732351 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732353 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732356 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732365 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732369 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.741190 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732371 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732374 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732377 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732386 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732390 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732393 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732396 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732398 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732401 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732404 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732407 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732410 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732412 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732415 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732417 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732420 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732423 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732426 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732429 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.741831 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732433 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732436 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732438 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732441 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732444 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732447 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732449 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732452 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732454 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732457 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732460 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732463 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732471 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732474 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732477 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732479 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732484 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732487 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732490 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.742294 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732499 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732502 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732505 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732508 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732511 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732514 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732517 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732520 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732522 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732525 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732528 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732531 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732534 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732538 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732541 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732543 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732546 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732549 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732552 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732554 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.742866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732557 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732560 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732562 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732565 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732567 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732570 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732579 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732581 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732584 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732587 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732589 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732592 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732594 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732597 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732600 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732603 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732605 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732607 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732610 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732612 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.743345 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732615 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732618 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732620 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732626 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732629 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732634 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732637 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732640 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732643 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732645 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732648 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732650 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732653 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732656 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732658 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.743824 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.732697 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732739 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732745 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732748 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732756 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732759 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732762 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732765 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732767 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732770 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732772 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732775 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732777 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732780 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732783 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732785 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732788 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732790 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732793 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732795 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.744207 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732798 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732802 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732805 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732808 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732810 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732813 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732816 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732818 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732821 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732823 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732826 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732828 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732831 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732834 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732836 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732839 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732841 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732849 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732852 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732854 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.744692 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732857 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732859 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732862 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732866 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732870 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732872 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732875 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732878 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732880 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732883 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732885 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732888 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732890 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732894 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732899 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732902 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732905 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732908 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732910 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.745174 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732913 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732915 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732918 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732921 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732923 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732926 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732928 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732931 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732934 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732937 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732939 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732949 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732951 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732954 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732957 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732959 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732962 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732964 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732967 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732970 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.745607 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732972 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732975 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732977 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732980 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732982 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732985 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732989 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732993 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732995 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.732998 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733001 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733004 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733006 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733009 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733012 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733014 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733017 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733020 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733023 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.746102 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733025 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733028 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733030 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733033 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733036 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733045 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733049 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733052 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.733056 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.740306 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.740320 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.740330 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.740335 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740542 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740568 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740574 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740579 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740591 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740595 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.746541 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740600 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740604 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740609 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740613 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740618 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740623 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740628 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740633 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740638 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740643 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740647 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740668 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740673 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740690 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740695 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740700 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740704 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740709 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740714 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740719 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.747088 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740724 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740730 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740736 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740747 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740751 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740755 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740760 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740764 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740768 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740772 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740777 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740781 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740786 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740790 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740794 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740799 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740807 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740812 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740816 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.747560 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740820 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740825 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740829 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740833 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740838 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740842 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740846 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740850 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740855 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740859 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740867 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740872 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740877 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740882 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740886 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740891 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740895 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740900 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740904 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.748024 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740909 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740913 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740918 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740927 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740931 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740936 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740940 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740944 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740948 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740953 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740957 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740961 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740966 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740970 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740974 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740978 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740987 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740991 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.740996 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.748513 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741000 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741005 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741009 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741013 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741017 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741021 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741026 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741032 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741036 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741045 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741050 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741055 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741059 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741063 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741067 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741071 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741078 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741084 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741089 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741093 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.748984 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741097 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741106 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741110 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.741166 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741763 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741776 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741780 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741783 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741786 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741788 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741794 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741797 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741801 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741803 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741806 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741809 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.749463 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741812 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741815 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741818 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741821 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741823 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741826 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741828 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741831 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741834 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741836 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741839 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741842 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741845 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741848 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741851 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741856 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741859 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741862 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741864 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.749879 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741867 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741869 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741873 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741876 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741879 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741882 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741884 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741887 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741889 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741894 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741897 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741900 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741903 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741906 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741908 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741911 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741913 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741916 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741919 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.750334 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741922 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741924 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741927 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741930 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741932 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741935 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741937 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741939 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741942 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741945 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741947 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741950 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741952 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741955 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741958 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741961 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741963 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741966 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741969 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741971 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.750885 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741974 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741977 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741979 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741982 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741985 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741987 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741990 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741994 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.741998 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742001 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742004 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742006 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742009 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742012 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742014 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742017 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742019 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742022 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742024 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.751378 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742027 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742029 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742032 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742034 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742037 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742039 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742045 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742048 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742051 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742054 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742056 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742059 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742063 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742067 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742070 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.751856 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742074 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.742105 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742141 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742147 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742151 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742153 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742156 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742159 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742162 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742165 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742168 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742171 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742173 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742176 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742179 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742181 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:40.752208 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742184 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742187 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742189 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742192 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742194 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742197 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742200 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742203 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742207 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742210 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742212 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742216 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742218 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742221 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742224 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742226 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742229 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742232 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742234 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:40.752597 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742237 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742240 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742243 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742246 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742249 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742252 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742255 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742257 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742260 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742262 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742265 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742267 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742270 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742272 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742275 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742277 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742280 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742289 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742292 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742295 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:40.753054 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742298 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742302 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742305 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742308 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742310 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742314 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742317 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742319 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742322 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742324 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742327 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742330 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742332 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742335 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742338 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742341 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742344 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742346 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742349 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742351 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:40.753516 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742354 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742356 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742359 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742361 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742364 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742368 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742372 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742375 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742378 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742381 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742383 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742386 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742388 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742392 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742395 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742397 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742400 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742403 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742405 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:40.754000 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742408 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742411 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742413 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742416 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742419 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742421 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742423 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742426 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742429 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742432 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742437 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742440 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742444 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:40.742447 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.742506 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.745196 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.746089 2582 server.go:1014] "Starting client certificate rotation" Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.746187 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:03:40.754433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.746222 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:03:40.754952 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.748730 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:03:40.754952 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.753248 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:03:40.762085 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.762067 2582 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:03:40.765562 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.765549 2582 log.go:25] "Validated CRI v1 image API" Oct 30 19:03:40.766331 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.766304 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:03:40.771076 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.771053 2582 fs.go:135] Filesystem UUIDs: map[38cf0db0-3fff-4b11-b529-8bb4ec7296cf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80095121-85be-4501-b4cb-1ab6d0e95113:/dev/nvme0n1p4] Oct 30 19:03:40.771162 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.771075 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:03:40.777139 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.777018 2582 manager.go:217] Machine: {Timestamp:2026-10-30 19:03:40.774924414 +0000 UTC m=+0.306872976 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101466 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27aaebdede8c2d37b6b579b69eb0bf SystemUUID:ec27aaeb-dede-8c2d-37b6-b579b69eb0bf BootID:5ba282a1-fa59-4efe-a293-af1affb5762d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:c4:03:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:c4:03:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:3e:41:a3:e5:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:03:40.777731 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.777717 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:03:40.777839 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.777824 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:03:40.780643 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780623 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:03:40.780789 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780646 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:03:40.780834 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780792 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:03:40.780834 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780799 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:03:40.780889 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780834 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:03:40.780889 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.780853 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:03:40.781967 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.781955 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:03:40.785454 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.785441 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:03:40.785569 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.785560 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:03:40.786262 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.786252 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:03:40.786297 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.786273 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:03:40.786336 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.786296 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:03:40.786336 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.786310 2582 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:03:40.786336 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.786319 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:03:40.787779 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.787752 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:03:40.789900 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.789880 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:03:40.796289 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.796273 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:03:40.797161 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797149 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:03:40.797212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797179 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:03:40.797356 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797346 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:03:40.797396 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797362 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:03:40.797396 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797368 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:03:40.797396 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797374 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:03:40.797396 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797380 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:03:40.797396 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797385 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:03:40.797956 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797948 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:03:40.797987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797957 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:03:40.797987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797965 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:03:40.797987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797971 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:03:40.797987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797980 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:03:40.797987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.797988 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:03:40.798661 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.798651 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:03:40.798661 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.798660 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:03:40.802346 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.802334 2582 server.go:1263] "Started kubelet" Oct 30 19:03:40.802467 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.802430 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:03:40.802520 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.802437 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:03:40.802563 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.802534 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:03:40.803069 ip-10-0-135-68 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:03:40.804168 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.804137 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:03:40.804955 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.804939 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:03:40.808749 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.808728 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:03:40.810119 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810095 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:03:40.810119 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810110 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:03:40.810271 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.810251 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found" Oct 30 19:03:40.810707 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810692 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:03:40.810812 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810763 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:03:40.810812 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810813 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:03:40.810976 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810902 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:03:40.810976 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.810910 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:03:40.811346 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811330 2582 factory.go:55] Registering systemd factory Oct 30 19:03:40.811411 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811366 2582 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:03:40.811582 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811567 2582 factory.go:153] Registering CRI-O factory Oct 30 19:03:40.811582 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811583 2582 factory.go:223] Registration of the crio container factory successfully Oct 30 19:03:40.811728 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811625 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:03:40.811728 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811644 2582 factory.go:103] Registering Raw factory Oct 30 19:03:40.811728 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.811698 2582 manager.go:1199] Started watching for new ooms in manager Oct 30 19:03:40.812739 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.812725 2582 manager.go:319] Starting recovery of all containers Oct 30 19:03:40.825387 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.825355 2582 manager.go:324] Recovery completed Oct 30 19:03:40.830047 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.830031 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:40.832261 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832235 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:40.832343 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832267 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:40.832343 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832277 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:40.832815 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832797 2582 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:03:40.832815 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832814 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:03:40.832928 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832825 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:40.832928 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.832834 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:03:40.834073 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834059 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:40.834186 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834082 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:40.834186 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834092 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:40.834250 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834208 2582 policy_none.go:50] "Start" Oct 30 19:03:40.834250 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834222 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:03:40.834250 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.834238 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:03:40.835384 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.835372 2582 policy_none.go:44] "Start" Oct 30 19:03:40.835754 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.835738 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:03:40.835828 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.835813 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nmn9b" Oct 30 19:03:40.836823 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.836802 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:03:40.837640 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.836790 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-68.ec2.internal.18e363fe5fb7bdb3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-68.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-68.ec2.internal,},FirstTimestamp:2026-10-30 19:03:40.802309555 +0000 UTC m=+0.334258118,LastTimestamp:2026-10-30 19:03:40.802309555 +0000 UTC m=+0.334258118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-68.ec2.internal,}" Oct 30 19:03:40.849590 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.849568 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:03:40.859055 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.859037 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:40.860260 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.860240 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:40.860321 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.860236 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:03:40.866102 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866090 2582 manager.go:341] "Starting Device Plugin manager" Oct 30 19:03:40.866157 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.866117 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:03:40.866157 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866125 2582 server.go:87] "Starting device plugin registration server" Oct 30 19:03:40.866342 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866318 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:03:40.866433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866343 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:03:40.866497 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866469 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:03:40.866701 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866616 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:03:40.866701 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.866627 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:03:40.866931 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.866910 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:03:40.866991 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.866959 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-68.ec2.internal\" not found" Oct 30 19:03:40.887078 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.887055 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:40.942489 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.942441 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:03:40.943758 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.943737 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:03:40.943854 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.943764 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:03:40.943854 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.943785 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:03:40.943854 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.943826 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:03:40.959771 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.959752 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:40.967202 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.967185 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:40.968166 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.968140 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:40.968259 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.968174 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:40.968259 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.968188 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:40.968259 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.968215 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-68.ec2.internal" Oct 30 19:03:40.988466 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:40.988413 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-68.ec2.internal" Oct 30 19:03:40.988466 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:40.988438 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-68.ec2.internal\": node \"ip-10-0-135-68.ec2.internal\" not found" Oct 30 19:03:41.044231 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.044076 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"] Oct 30 19:03:41.045689 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.045667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.046598 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.046581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.064221 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.064202 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.067018 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.067003 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.090862 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.090842 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:03:41.094723 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.094709 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:03:41.112239 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.112217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.112327 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.112247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.112327 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.112274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212513 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.212655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.212627 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.397858 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.397756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.399882 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.399858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:03:41.745920 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.745854 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:03:41.746355 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.745982 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:03:41.746355 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.746014 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:03:41.746355 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:41.746016 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:03:42.895119 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.895081 2582 apiserver.go:52] "Watching apiserver" Oct 30 19:03:42.918580 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.918558 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:03:42.945803 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.945778 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:03:42.946104 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.946077 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-l9rzn","openshift-network-operator/iptables-alerter-lzg24","kube-system/global-pull-secret-syncer-9kwwp","openshift-cluster-node-tuning-operator/tuned-fkzfj","openshift-dns/node-resolver-s4kdn","openshift-multus/multus-additional-cni-plugins-6txgd","openshift-network-diagnostics/network-check-target-nhhzq","openshift-ovn-kubernetes/ovnkube-node-8mt7n","openshift-image-registry/node-ca-zhtch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","openshift-multus/multus-p2579","kube-system/konnectivity-agent-5cp96","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj"] Oct 30 19:03:42.947235 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.947210 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:41 +0000 UTC" deadline="2028-06-13 21:03:31.07797206 +0000 UTC" Oct 30 19:03:42.947235 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.947234 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14209h59m48.13074113s" Oct 30 19:03:42.947846 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.947823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:42.947939 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:42.947918 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:42.948950 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.948933 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:42.950138 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.950119 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:42.950235 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:42.950179 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:42.951878 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.951859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:42.953001 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.952979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:42.954463 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.954449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:42.955873 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.955858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:42.955947 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:42.955922 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:42.957559 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.957545 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:42.960174 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.960150 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:42.962101 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.962068 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2579" Oct 30 19:03:42.964169 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.964151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:42.966463 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.966441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:42.967997 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.967980 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:42.969273 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.969258 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\"" Oct 30 19:03:42.969273 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.969270 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\"" Oct 30 19:03:42.969353 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.969325 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\"" Oct 30 19:03:42.971075 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:42.971061 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\"" Oct 30 19:03:43.019244 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.019226 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:03:43.023135 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.023219 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5nk\" (UniqueName: \"kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.023219 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.023309 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrgr\" (UniqueName: \"kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.023355 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnc4\" (UniqueName: \"kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.023402 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:43.023452 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.023504 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.023504 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.023591 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023521 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.023591 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.023591 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplfk\" (UniqueName: \"kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.023709 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.023893 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2hs\" (UniqueName: \"kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.023893 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.023893 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.023893 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.023893 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.024037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.023981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v998c\" (UniqueName: \"kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024204 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024404 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq695\" (UniqueName: \"kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024601 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.024714 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024750 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chshm\" (UniqueName: \"kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.024990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.025034 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.025490 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.025262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.037824 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.037788 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e8abc4535060f5a3a432c3ab1497e7.slice/crio-49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37 WatchSource:0}: Error finding container 49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37: Status 404 returned error can't find the container with id 49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37 Oct 30 19:03:43.038170 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.038148 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a30bdfa789f3bccffc4563d5bc301a.slice/crio-ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e WatchSource:0}: Error finding container ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e: Status 404 returned error can't find the container with id ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e Oct 30 19:03:43.044486 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.044468 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:03:43.054523 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.054458 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"c1a30bdfa789f3bccffc4563d5bc301a","Type":"ContainerStarted","Data":"ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e"} Oct 30 19:03:43.055685 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.055662 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerStarted","Data":"49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37"} Oct 30 19:03:43.062522 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.062502 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:03:43.062603 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.062562 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k8mdp" Oct 30 19:03:43.068229 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068212 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:03:43.068229 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068221 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.068406 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068392 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:03:43.068488 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068474 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.068616 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068605 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:03:43.068792 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068771 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.068919 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068801 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.068919 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068836 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:03:43.068919 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068909 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.069082 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068918 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:03:43.069082 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068942 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.069082 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068979 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.069082 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.068993 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069129 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069135 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069139 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069158 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069192 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069194 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069205 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:03:43.069295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069249 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.069507 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069301 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:03:43.069705 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069689 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:03:43.069766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069703 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:03:43.069766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069710 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\"" Oct 30 19:03:43.069766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069690 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:03:43.069766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069738 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:03:43.069766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.069734 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\"" Oct 30 19:03:43.076826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.076808 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\"" Oct 30 19:03:43.082735 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.082718 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\"" Oct 30 19:03:43.083426 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.083394 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\"" Oct 30 19:03:43.125905 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.125905 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.125905 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.125905 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5nk\" (UniqueName: \"kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrgr\" (UniqueName: \"kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125959 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125971 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnc4\" (UniqueName: \"kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125967 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.125997 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126180 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.126212 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplfk\" (UniqueName: \"kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126271 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.126405 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126419 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.126516 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126580 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.126587 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.626548444 +0000 UTC m=+2.051084259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126595 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126657 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.126783 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.126683 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.626672138 +0000 UTC m=+2.051207929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126712 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2hs\" (UniqueName: \"kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126799 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126825 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126871 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126852 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126944 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.126987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.127605 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127058 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127115 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127155 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127158 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127178 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v998c\" (UniqueName: \"kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127349 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127430 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.128392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127488 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127500 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127508 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127540 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq695\" (UniqueName: \"kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127607 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127625 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127500 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127547 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127655 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127687 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127724 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.129199 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127778 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127878 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127872 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127892 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127948 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127971 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127985 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.127898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128006 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128103 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128109 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130068 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128287 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128329 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128348 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128399 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128409 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128412 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128430 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128634 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128772 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128806 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.128583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129073 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.130704 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129162 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129201 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chshm\" (UniqueName: \"kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129116 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129282 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129557 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.131203 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129639 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129657 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129721 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129709 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129882 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.129922 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.130026 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.130068 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.130086 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.130160 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.130372 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.131492 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.131570 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.131694 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.131826 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.131772 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.146718 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.146696 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnc4\" (UniqueName: \"kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.146853 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.146746 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:43.146853 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.146768 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:43.146853 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.146780 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.146853 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.146827 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.646812512 +0000 UTC m=+2.071348307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.148020 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.147981 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5nk\" (UniqueName: \"kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:03:43.148197 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.148157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2hs\" (UniqueName: \"kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.148432 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.148414 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplfk\" (UniqueName: \"kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.149844 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.149818 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrgr\" (UniqueName: \"kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.149940 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.149824 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v998c\" (UniqueName: \"kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.150051 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.150033 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq695\" (UniqueName: \"kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.150439 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.150420 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.150476 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.150441 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chshm\" (UniqueName: \"kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.283452 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.283423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:03:43.293141 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.293118 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61da23fa_4dc5_4973_af62_c338ecdff26d.slice/crio-329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f WatchSource:0}: Error finding container 329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f: Status 404 returned error can't find the container with id 329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f Oct 30 19:03:43.302801 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.302772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:03:43.311733 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.311711 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef107f8_59da_48f3_81dc_df4851b26044.slice/crio-ea36d89a18ef464acbaf696dc64b358bfc33a367447541e52bcd152880515223 WatchSource:0}: Error finding container ea36d89a18ef464acbaf696dc64b358bfc33a367447541e52bcd152880515223: Status 404 returned error can't find the container with id ea36d89a18ef464acbaf696dc64b358bfc33a367447541e52bcd152880515223 Oct 30 19:03:43.319534 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.319515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:03:43.327485 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.327465 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b7c1f1_3b2d_4b73_8870_f1cffe60206e.slice/crio-18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f WatchSource:0}: Error finding container 18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f: Status 404 returned error can't find the container with id 18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f Oct 30 19:03:43.363653 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.363631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:03:43.365791 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.365774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:03:43.369433 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.369415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2579" Oct 30 19:03:43.373849 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.373830 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e6381_5d5c_4d33_86cb_3e528c197bcd.slice/crio-56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb WatchSource:0}: Error finding container 56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb: Status 404 returned error can't find the container with id 56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb Oct 30 19:03:43.375326 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.375311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:43.376396 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.376375 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9d8a4b_34d0_47ba_abc1_8e71a55d600c.slice/crio-00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7 WatchSource:0}: Error finding container 00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7: Status 404 returned error can't find the container with id 00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7 Oct 30 19:03:43.379236 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.379219 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67705ac6_bf6a_4e9e_9749_691459286cde.slice/crio-e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578 WatchSource:0}: Error finding container e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578: Status 404 returned error can't find the container with id e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578 Oct 30 19:03:43.385151 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.385134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:03:43.385268 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.385249 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2c4560_125c_4c15_a15e_9e93a59e9a06.slice/crio-2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75 WatchSource:0}: Error finding container 2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75: Status 404 returned error can't find the container with id 2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75 Oct 30 19:03:43.389936 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.389920 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:03:43.394390 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.394371 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode066ed65_096a_4d83_bcf0_cc3cb10267fc.slice/crio-d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a WatchSource:0}: Error finding container d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a: Status 404 returned error can't find the container with id d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a Oct 30 19:03:43.398433 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:03:43.398415 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd9e114_e938_4928_94e8_c9c3bd7ae37b.slice/crio-fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f WatchSource:0}: Error finding container fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f: Status 404 returned error can't find the container with id fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f Oct 30 19:03:43.633801 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.633713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:43.633801 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.633755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:43.634003 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.633871 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.634003 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.633922 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:03:44.633909753 +0000 UTC m=+3.058445559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.634336 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.634243 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.634336 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.634308 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:44.634291233 +0000 UTC m=+3.058827040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.721320 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.721245 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:43.735378 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.734799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:43.735378 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.734938 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:43.735378 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.734960 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:43.735378 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.734972 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.735378 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:43.735028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:44.735010328 +0000 UTC m=+3.159546124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.897968 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:43.897898 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:44.065702 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.065662 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f"} Oct 30 19:03:44.065931 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.065887 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:42 +0000 UTC" deadline="2028-07-11 13:51:43.078556618 +0000 UTC" Oct 30 19:03:44.065931 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.065932 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14874h47m59.012628131s" Oct 30 19:03:44.094035 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.093772 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhtch" event={"ID":"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c","Type":"ContainerStarted","Data":"00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7"} Oct 30 19:03:44.111020 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.110937 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerStarted","Data":"56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb"} Oct 30 19:03:44.123351 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.123280 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" event={"ID":"e066ed65-096a-4d83-bcf0-cc3cb10267fc","Type":"ContainerStarted","Data":"d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a"} Oct 30 19:03:44.126225 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.126076 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5cp96" event={"ID":"cd2c4560-125c-4c15-a15e-9e93a59e9a06","Type":"ContainerStarted","Data":"2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75"} Oct 30 19:03:44.139180 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.139044 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2579" event={"ID":"67705ac6-bf6a-4e9e-9749-691459286cde","Type":"ContainerStarted","Data":"e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578"} Oct 30 19:03:44.147347 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.147323 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4kdn" event={"ID":"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e","Type":"ContainerStarted","Data":"18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f"} Oct 30 19:03:44.162906 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.162810 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" event={"ID":"6ef107f8-59da-48f3-81dc-df4851b26044","Type":"ContainerStarted","Data":"ea36d89a18ef464acbaf696dc64b358bfc33a367447541e52bcd152880515223"} Oct 30 19:03:44.164841 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.164815 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzg24" event={"ID":"61da23fa-4dc5-4973-af62-c338ecdff26d","Type":"ContainerStarted","Data":"329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f"} Oct 30 19:03:44.218837 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.218638 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:44.640865 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.640567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:44.640865 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.640613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:44.640865 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.640743 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:44.640865 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.640796 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:03:46.640779668 +0000 UTC m=+5.065315478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:44.640865 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.640859 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:44.641235 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.640962 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:46.640938037 +0000 UTC m=+5.065473844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:44.741575 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:44.741538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:44.741801 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.741785 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:44.741860 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.741809 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:44.741860 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.741822 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:44.741958 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:44.741882 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:46.741862344 +0000 UTC m=+5.166398161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:45.052001 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:45.051922 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:45.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:45.052052 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:45.052409 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:45.052182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:45.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:45.052281 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:45.052409 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:45.052305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:45.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:45.052381 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:45.066438 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:45.066402 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:42 +0000 UTC" deadline="2028-06-10 16:59:07.887215723 +0000 UTC" Oct 30 19:03:45.066438 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:45.066435 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14133h55m22.820785509s" Oct 30 19:03:46.658045 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:46.658001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:46.658655 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:46.658067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:46.658655 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.658227 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:46.658655 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.658294 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.658274809 +0000 UTC m=+9.082810603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:46.658821 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.658660 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:46.658821 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.658695 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.658683327 +0000 UTC m=+9.083219118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:46.759602 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:46.758892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:46.759602 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.759114 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:46.759602 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.759134 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:46.759602 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.759146 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:46.759602 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:46.759205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.759187135 +0000 UTC m=+9.183722941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:47.052493 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.052256 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:47.052667 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:47.052638 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:47.053048 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.053024 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:47.053180 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:47.053158 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:47.053241 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.053220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:47.053311 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:47.053292 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:47.206591 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.205648 2582 generic.go:356] "Generic (PLEG): container finished" podID="a5e8abc4535060f5a3a432c3ab1497e7" containerID="4c19d7d083650e75f8f19183fe67292a5398806c46907729b52d75935ceae8f7" exitCode=0 Oct 30 19:03:47.206591 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.205739 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerDied","Data":"4c19d7d083650e75f8f19183fe67292a5398806c46907729b52d75935ceae8f7"} Oct 30 19:03:47.218418 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.218391 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="7e233076edb60ca7da6ad19e4d929322018faf34644f2bd067a7a2a782c6998e" exitCode=0 Oct 30 19:03:47.218521 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:47.218438 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"7e233076edb60ca7da6ad19e4d929322018faf34644f2bd067a7a2a782c6998e"} Oct 30 19:03:48.226550 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:48.226473 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"c1a30bdfa789f3bccffc4563d5bc301a","Type":"ContainerStarted","Data":"8f75cc509fd88386aef1957f9f6549418d05caa0d891ed603d18cdc240b9dfb1"} Oct 30 19:03:48.233465 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:48.233423 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerStarted","Data":"b4a41e8be2b2215217bb8210cf176a4dd5a6369ac5095013295f45185c94356c"} Oct 30 19:03:48.237855 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:48.237439 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5cp96" event={"ID":"cd2c4560-125c-4c15-a15e-9e93a59e9a06","Type":"ContainerStarted","Data":"50493f371acb2ed2300eece599ba600f26116bd818f6e827d339fa87ec23f84d"} Oct 30 19:03:48.268463 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:48.267982 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podStartSLOduration=6.26796351 podStartE2EDuration="6.26796351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:48.267049329 +0000 UTC m=+6.691585145" watchObservedRunningTime="2026-10-30 19:03:48.26796351 +0000 UTC m=+6.692499324" Oct 30 19:03:48.351761 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:48.351191 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" podStartSLOduration=6.3511723589999995 podStartE2EDuration="6.351172359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:48.309058055 +0000 UTC m=+6.733593869" watchObservedRunningTime="2026-10-30 19:03:48.351172359 +0000 UTC m=+6.775708176" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:49.051835 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:49.051989 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:49.052391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:49.052488 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:49.052556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:49.052678 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:49.052626 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:49.243502 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:49.242719 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhtch" event={"ID":"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c","Type":"ContainerStarted","Data":"0097a7f9a1161ed36722f9d0ff53eb746ad492a4786c57d4b73f16e7a1df1882"} Oct 30 19:03:49.284860 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:49.283683 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5cp96" podStartSLOduration=3.058987193 podStartE2EDuration="7.283664129s" totalImagesPullingTime="4.224676936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:48.351742627 +0000 UTC m=+6.776278443" watchObservedRunningTime="2026-10-30 19:03:49.283664129 +0000 UTC m=+7.708199949" Oct 30 19:03:50.248898 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:50.248672 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" event={"ID":"e066ed65-096a-4d83-bcf0-cc3cb10267fc","Type":"ContainerStarted","Data":"bd2fa3a3c29fc396b79e01fc494870d3f1293703b95cd3e798a67720f9ecbfae"} Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:50.693248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:50.693298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.693466 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.693533 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.693515983 +0000 UTC m=+17.118051799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.693601 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:50.693990 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.693636 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.693625028 +0000 UTC m=+17.118160822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:50.795136 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:50.794525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:50.795136 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.794752 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:50.795136 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.794772 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:50.795136 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.794786 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:50.795136 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:50.794846 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.794828349 +0000 UTC m=+17.219364163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.051587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:51.051725 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.052125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:51.052216 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.052261 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:51.052409 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:51.052320 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:51.478632 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.478597 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:51.479305 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.478957 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:03:51.479581 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.479559 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:03:51.522119 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.519759 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zhtch" podStartSLOduration=4.518859751 podStartE2EDuration="9.51973794s" totalImagesPullingTime="5.000878189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:49.284822324 +0000 UTC m=+7.709358140" watchObservedRunningTime="2026-10-30 19:03:51.51973794 +0000 UTC m=+9.944273756" Oct 30 19:03:51.977335 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.977202 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:03:51.478969452Z","UUID":"9ae2547b-a749-414e-b7e0-0d6176f96283","Handler":null,"Name":"","Endpoint":""} Oct 30 19:03:51.979261 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.979233 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:03:51.979411 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:51.979396 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:03:52.254346 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.254107 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="24c1dfd33bec0a8b8d84c1a484124e3c205f285b7174c3244a2bc0c77de67f1f" exitCode=0 Oct 30 19:03:52.254585 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.254560 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"24c1dfd33bec0a8b8d84c1a484124e3c205f285b7174c3244a2bc0c77de67f1f"} Oct 30 19:03:52.260210 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.260184 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" event={"ID":"e066ed65-096a-4d83-bcf0-cc3cb10267fc","Type":"ContainerStarted","Data":"7414e490dee98be489eaadb1e739c0ce38a66203a1ccca617fc66378ae3bb8e1"} Oct 30 19:03:52.262377 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.262351 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2579" event={"ID":"67705ac6-bf6a-4e9e-9749-691459286cde","Type":"ContainerStarted","Data":"1b0442dbd52d72ee5da913ca32b9616a81ad3ec269c6f4d0b2659625eeb8f7fb"} Oct 30 19:03:52.266667 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.266599 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4kdn" event={"ID":"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e","Type":"ContainerStarted","Data":"8a0a8c10614771d6c057759e238900230fb814cc606b8c9b1e2c3d99c58f214c"} Oct 30 19:03:52.343848 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:52.343775 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s4kdn" podStartSLOduration=2.517461574 podStartE2EDuration="10.343733698s" totalImagesPullingTime="7.826272124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.343459405 +0000 UTC m=+10.767995220" watchObservedRunningTime="2026-10-30 19:03:52.343733698 +0000 UTC m=+10.768269503" Oct 30 19:03:53.051860 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.051824 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:53.052325 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.051873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:53.052325 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.051874 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:53.052325 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:53.051947 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:53.052325 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:53.052030 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:53.052325 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:53.052127 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:53.270710 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.270630 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzg24" event={"ID":"61da23fa-4dc5-4973-af62-c338ecdff26d","Type":"ContainerStarted","Data":"f2f2e9e14e89643038b0b6d4b70c61e7e37447ece1b71df61f05f5cde13c12e2"} Oct 30 19:03:53.273384 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.273353 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="98bc8f17fe31d3558a1d8e280c974c146c592ba309943ab7cfe7df3b887c97e8" exitCode=0 Oct 30 19:03:53.273517 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.273449 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"98bc8f17fe31d3558a1d8e280c974c146c592ba309943ab7cfe7df3b887c97e8"} Oct 30 19:03:53.276495 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.276468 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" event={"ID":"e066ed65-096a-4d83-bcf0-cc3cb10267fc","Type":"ContainerStarted","Data":"12bbf0ffa87de641c34fc1814045a12f0fc06469ca010f9efe5a99cb00f8e739"} Oct 30 19:03:53.316243 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.316186 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-p2579" podStartSLOduration=3.293158304 podStartE2EDuration="11.316168833s" totalImagesPullingTime="8.023010529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.391616889 +0000 UTC m=+10.816152701" watchObservedRunningTime="2026-10-30 19:03:53.316168833 +0000 UTC m=+11.740704648" Oct 30 19:03:53.316625 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.316591 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lzg24" podStartSLOduration=3.211329751 podStartE2EDuration="11.316583599s" totalImagesPullingTime="8.105253848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:53.3162755 +0000 UTC m=+11.740811315" watchObservedRunningTime="2026-10-30 19:03:53.316583599 +0000 UTC m=+11.741119412" Oct 30 19:03:53.404224 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:53.404177 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" podStartSLOduration=2.682028371 podStartE2EDuration="11.404162802s" totalImagesPullingTime="8.722134431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:53.404072038 +0000 UTC m=+11.828607852" watchObservedRunningTime="2026-10-30 19:03:53.404162802 +0000 UTC m=+11.828698613" Oct 30 19:03:54.279742 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:54.279656 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" event={"ID":"6ef107f8-59da-48f3-81dc-df4851b26044","Type":"ContainerStarted","Data":"9492c3678ac24c2ff6ac29a2054ba78b889d3621898727ac3a78815a41865de6"} Oct 30 19:03:54.282636 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:54.282600 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="fb51161ab52b63a158a48570bf72d15351e3d87bfce2efd7b0f2874cb9cacb64" exitCode=0 Oct 30 19:03:54.282766 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:54.282694 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"fb51161ab52b63a158a48570bf72d15351e3d87bfce2efd7b0f2874cb9cacb64"} Oct 30 19:03:54.323715 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:54.323666 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" podStartSLOduration=2.149910315 podStartE2EDuration="12.32365257s" totalImagesPullingTime="10.173742255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:54.32321858 +0000 UTC m=+12.747754394" watchObservedRunningTime="2026-10-30 19:03:54.32365257 +0000 UTC m=+12.748188389" Oct 30 19:03:55.052411 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:55.052374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:55.052594 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:55.052424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:55.052594 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:55.052437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:55.052594 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:55.052496 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:55.052594 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:55.052554 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:55.052749 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:55.052637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:56.136927 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:56.136892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:56.291739 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:56.291071 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerStarted","Data":"38493811cf192e9d5ae0f3b3624e54784cdc12a5333dd31f8f331ee2d88c7f68"} Oct 30 19:03:56.336325 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:56.336290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:57.052250 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.052218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:57.052399 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.052218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:57.052399 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:57.052335 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:57.052510 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.052218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:57.052510 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:57.052429 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:57.052510 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:57.052479 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:57.296909 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.296871 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="38493811cf192e9d5ae0f3b3624e54784cdc12a5333dd31f8f331ee2d88c7f68" exitCode=0 Oct 30 19:03:57.296909 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.296917 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"38493811cf192e9d5ae0f3b3624e54784cdc12a5333dd31f8f331ee2d88c7f68"} Oct 30 19:03:57.734630 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.734552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:57.878945 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:57.878919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:58.301183 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301166 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:03:58.301907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301489 2582 generic.go:356] "Generic (PLEG): container finished" podID="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" containerID="779d85a2a2c767a2aa2b41961fdc28c9ff16aeb1610b3a4a3d3cf86e2c4e1111" exitCode=1 Oct 30 19:03:58.301907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301554 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"dd142e4ebe425e813eb75bff06013a96d568e2403fc9dcbc6be81d75b0ea000d"} Oct 30 19:03:58.301907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301619 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"4714fca2f3e6669b11caad6546aba58a23d56a01bbfeff1695c58d61f6fbc27f"} Oct 30 19:03:58.301907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301662 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerDied","Data":"779d85a2a2c767a2aa2b41961fdc28c9ff16aeb1610b3a4a3d3cf86e2c4e1111"} Oct 30 19:03:58.301907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.301703 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"3551197bb0ffd0710d7a341b81767d431be04da803f364cb3d8f1ba5688c338f"} Oct 30 19:03:58.304936 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.304918 2582 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="0aa7793bea0325f2d22c7aece8f662b2a6db555d8eb3c3f71788af1e3452ba8d" exitCode=0 Oct 30 19:03:58.305030 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.304960 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerDied","Data":"0aa7793bea0325f2d22c7aece8f662b2a6db555d8eb3c3f71788af1e3452ba8d"} Oct 30 19:03:58.758734 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.758558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:58.758923 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.758749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:58.758923 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.758706 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:58.758923 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.758836 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:58.758923 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.758857 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:14.758841486 +0000 UTC m=+33.183377295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:58.758923 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.758874 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:04:14.758863835 +0000 UTC m=+33.183399627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:58.858994 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:58.858968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:58.859171 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.859155 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:58.859233 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.859179 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:58.859233 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.859193 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:58.859345 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:58.859255 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:14.859235586 +0000 UTC m=+33.283771379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:59.052324 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.052244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:03:59.052455 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.052327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:03:59.052455 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.052346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:03:59.052455 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:59.052447 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:03:59.052556 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:59.052518 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:03:59.052586 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:03:59.052567 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:03:59.309470 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.309406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:03:59.309839 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.309747 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"7b38eed956d9d19c04c85624914c46c675f91f3127058b0d7cd043ad598d81a2"} Oct 30 19:03:59.309839 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.309813 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"df20dea1793cfef464e5d437e0e77ea4ea14a0a18bbeae6bd473df59653aa089"} Oct 30 19:03:59.312824 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.312800 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6txgd" event={"ID":"5d5e6381-5d5c-4d33-86cb-3e528c197bcd","Type":"ContainerStarted","Data":"685f2e08165349b8a3da876babd1802eb14d3c748ce1a34821f27f425c0f938e"} Oct 30 19:03:59.387508 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.387459 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6txgd" podStartSLOduration=7.000538283 podStartE2EDuration="17.387446661s" totalImagesPullingTime="10.386908378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:59.387179701 +0000 UTC m=+17.811715515" watchObservedRunningTime="2026-10-30 19:03:59.387446661 +0000 UTC m=+17.811982475" Oct 30 19:03:59.430007 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.429974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:59.561518 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:03:59.561438 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.051732 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.051697 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:01.051732 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.051726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:01.052345 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:01.051829 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:01.052345 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.051857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:01.052345 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:01.051931 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:01.052345 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:01.052000 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:01.189987 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.189934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.320186 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.320115 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:01.320455 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.320436 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"7352a48bb8d83afdaea1ebd7e8fb8dc240f24af84b3ce525f5d4a9c05d356356"} Oct 30 19:04:01.348264 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.348236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.522874 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.522839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.702771 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.702685 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.867491 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:01.867462 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.006903 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.006833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.157489 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.157456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.537913 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.537883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.674577 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.674546 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:02.674738 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.674688 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:02.675178 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.675154 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:02.898331 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:02.898242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.051827 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.051794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:03.051827 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.051820 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:03.052031 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.051931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:03.052031 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:03.051936 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:03.052175 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:03.052040 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:03.052175 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:03.052152 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:03.053415 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.053370 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.327874 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.327785 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:03.328558 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.328137 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"2cf18feb2a30898002aeb05685e06dd96b161145d8ab07d8949f37bc72e07d3f"} Oct 30 19:04:03.328740 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.328500 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:03.328802 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.328561 2582 scope.go:122] "RemoveContainer" containerID="779d85a2a2c767a2aa2b41961fdc28c9ff16aeb1610b3a4a3d3cf86e2c4e1111" Oct 30 19:04:03.344179 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.344162 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:03.352014 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.351987 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.511715 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.511688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.646314 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.646199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.811538 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.811507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.957421 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:03.957332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:04.336412 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.336331 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:04.336775 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.336669 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" event={"ID":"4fd9e114-e938-4928-94e8-c9c3bd7ae37b","Type":"ContainerStarted","Data":"7fa92d20bedb39dcc4575671b0503ccc3b2f78f976e32720cc833b5f0d3d8948"} Oct 30 19:04:04.336775 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.336755 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:04.337024 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.337004 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:04.351371 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.351335 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:04.431833 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:04.431782 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" podStartSLOduration=7.897127159 podStartE2EDuration="22.431769785s" totalImagesPullingTime="14.534642626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:04.431396174 +0000 UTC m=+22.855931990" watchObservedRunningTime="2026-10-30 19:04:04.431769785 +0000 UTC m=+22.856305598" Oct 30 19:04:05.036513 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.036482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.052313 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.052291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:05.052444 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.052384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:05.052444 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:05.052390 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:05.052444 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.052401 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:05.052545 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:05.052474 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:05.052577 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:05.052550 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:05.242754 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.242722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.339511 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.339441 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:05.386381 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.386353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.528825 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.528794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.665666 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.665582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.824489 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:05.824457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:06.341871 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:06.341845 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:06.647233 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:06.647164 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.027403 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.027326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.052362 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.052341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:07.052462 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.052448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:07.052508 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:07.052461 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:07.052552 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:07.052523 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:07.052552 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.052537 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:07.052637 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:07.052621 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:07.164391 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.164362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.372585 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.372554 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.514004 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.513974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.650812 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.650740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.822782 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:07.822740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:08.269275 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:08.269246 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4kdn_e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/dns-node-resolver/0.log" Oct 30 19:04:09.027701 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.027661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.051764 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.051746 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:09.051840 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.051829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:09.051880 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.051859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:09.051947 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:09.051929 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:09.052008 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:09.051989 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:09.052082 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:09.052062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:09.171022 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.170994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.316376 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.316341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.455054 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.455030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.469160 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.469140 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhtch_8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/node-ca/0.log" Oct 30 19:04:09.633157 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.633055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.822624 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:09.822575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:10.727741 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:10.727711 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:10.728137 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:10.727957 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:10.737259 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:10.737210 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" podUID="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:04:10.746282 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:10.746253 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" podUID="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:04:11.035903 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.035820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.051948 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.051924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:11.052067 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.051924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:11.052067 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:11.052022 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:11.052189 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.051924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:11.052189 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:11.052129 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:11.052252 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:11.052237 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:11.176936 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.176904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.321360 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.321334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.456623 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.456593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.630057 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.629990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.836507 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:11.836484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.017087 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.017050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.052149 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.052124 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:13.052261 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.052132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:13.052261 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.052232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:13.052261 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:13.052235 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:13.052366 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:13.052299 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:13.052366 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:13.052350 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:13.230134 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.230080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.456194 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.456165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.644316 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.644288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.830629 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:13.830592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:14.230026 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:14.229946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:14.777134 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:14.777080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:14.777134 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:14.777132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:14.777542 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.777224 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:04:14.777542 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.777226 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:04:14.777542 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.777273 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret podName:fc2715ff-fc4b-46c6-89fe-f95275cfe46d nodeName:}" failed. No retries permitted until 2026-10-30 19:04:46.777259561 +0000 UTC m=+65.201795366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret") pod "global-pull-secret-syncer-9kwwp" (UID: "fc2715ff-fc4b-46c6-89fe-f95275cfe46d") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:04:14.777542 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.777286 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs podName:ed069dac-23ef-4ec7-88a9-e74dc4e4161b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:46.77728067 +0000 UTC m=+65.201816462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs") pod "network-metrics-daemon-l9rzn" (UID: "ed069dac-23ef-4ec7-88a9-e74dc4e4161b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:04:14.877562 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:14.877533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:14.877700 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.877642 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:04:14.877700 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.877655 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:04:14.877700 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.877664 2582 projected.go:196] Error preparing data for projected volume kube-api-access-zrgs6 for pod openshift-network-diagnostics/network-check-target-nhhzq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:04:14.877798 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:14.877704 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6 podName:e8b22f35-75f3-4149-a2aa-5ee5074fa474 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:46.87769209 +0000 UTC m=+65.302227894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zrgs6" (UniqueName: "kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6") pod "network-check-target-nhhzq" (UID: "e8b22f35-75f3-4149-a2aa-5ee5074fa474") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:04:15.051715 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:15.051687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:15.051715 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:15.051712 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:15.051907 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:15.051812 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:15.051907 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:15.051865 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:15.051979 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:15.051945 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:15.052033 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:15.052016 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:17.051456 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:17.051425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:17.051456 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:17.051450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:17.051917 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:17.051527 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:17.051917 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:17.051574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:17.051917 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:17.051642 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:17.051917 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:17.051705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:19.051939 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:19.051910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:19.051939 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:19.051929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:19.052483 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:19.051909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:19.052483 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:19.052032 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:19.052483 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:19.052103 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:19.052483 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:19.052140 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:19.967751 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:19.967721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.104286 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.104250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.240829 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.240759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.376079 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.376049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.516776 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.516704 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.694285 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.694257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.833515 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:20.833489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:21.051702 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:21.051672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:21.051868 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:21.051780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:21.051868 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:21.051794 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:21.051951 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:21.051894 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:21.052013 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:21.051997 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:21.052117 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:21.052087 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:23.051999 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:23.051967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:23.052371 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:23.051968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:23.052371 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:23.052062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:23.052371 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:23.051967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:23.052371 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:23.052199 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:23.052371 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:23.052255 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:23.982472 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:23.982437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:24.160964 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:24.160931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:24.302252 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:24.302227 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:25.052221 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.052188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:25.052467 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.052262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:25.052467 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.052287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:25.052467 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.052360 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:25.052467 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.052402 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:25.052467 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.052458 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:25.606214 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.606027 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9rzn"] Oct 30 19:04:25.606573 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.606292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:25.606573 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.606378 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:25.608925 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.608896 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhhzq"] Oct 30 19:04:25.609052 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.608999 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:25.609137 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.609116 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:25.609369 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.609352 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kwwp"] Oct 30 19:04:25.609429 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:25.609420 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:25.609500 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:25.609485 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:26.550860 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:26.550832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:26.703315 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:26.703280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:27.052213 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:27.052188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:27.052350 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:27.052317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:27.052350 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:27.052337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:27.052415 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:27.052342 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:27.052447 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:27.052416 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:27.052499 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:27.052484 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:27.287842 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:27.287816 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4kdn_e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/dns-node-resolver/0.log" Oct 30 19:04:28.679733 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:28.679707 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhtch_8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/node-ca/0.log" Oct 30 19:04:29.051813 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.051789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:29.051979 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.051879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:29.051979 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:29.051886 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:29.051979 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.051892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:29.051979 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:29.051956 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:29.052224 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:29.052025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:29.477860 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.477788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:29.611958 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.611927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:29.964201 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:29.964170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:30.987684 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:30.987652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:31.051875 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.051847 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:31.051875 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.051877 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:31.052081 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.051894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:31.052081 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:31.052017 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:31.052212 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:31.052140 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:31.052269 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:31.052220 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:31.129134 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.129106 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:31.255043 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.254981 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z5m5f"] Oct 30 19:04:31.258428 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.258406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.282990 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.282966 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:04:31.284307 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.284284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:31.291896 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.291878 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sz8sw\"" Oct 30 19:04:31.291896 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.291878 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:04:31.364031 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.364007 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:04:31.364163 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.364142 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:04:31.364163 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.364145 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:04:31.364247 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.364214 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:04:31.400868 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.400846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.400980 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.400876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.400980 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.400897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.400980 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.400914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.401112 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.400984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.401112 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.401057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.401192 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.401112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.401192 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.401155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.401192 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.401182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jc45\" (UniqueName: \"kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.420936 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.420917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:31.501741 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501847 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501847 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501847 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.501981 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.501977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc45\" (UniqueName: \"kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.502299 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.502011 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.502299 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.502038 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.502299 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.502196 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.502426 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.502404 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.502609 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.502587 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.505877 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.505820 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.505877 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.505867 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.522182 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.522164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc45\" (UniqueName: \"kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.571079 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.571058 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:31.591729 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.591672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:31.724792 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:31.724763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.360853 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:32.360820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.398376 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:32.398340 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5m5f" event={"ID":"53bc3361-f8d3-49de-8490-baf98084ac36","Type":"ContainerStarted","Data":"079acf535272fe24b9ebd036cc8c3d186aeeb53bab85003d7929437784c751f4"} Oct 30 19:04:32.498086 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:32.498019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.632459 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:32.632341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.775220 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:32.775191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:33.051390 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.051367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:33.051546 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.051390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:33.051546 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.051413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:33.051546 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:33.051481 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:33.051654 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:33.051555 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:33.051654 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:33.051620 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:33.128803 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.128777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:33.283521 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.283487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:33.401825 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.401759 2582 generic.go:356] "Generic (PLEG): container finished" podID="53bc3361-f8d3-49de-8490-baf98084ac36" containerID="f50a31a37ba3d733351e3026bf77a7b46a9e4daef3e56573068d128ce0aa353b" exitCode=0 Oct 30 19:04:33.401825 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:33.401805 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5m5f" event={"ID":"53bc3361-f8d3-49de-8490-baf98084ac36","Type":"ContainerDied","Data":"f50a31a37ba3d733351e3026bf77a7b46a9e4daef3e56573068d128ce0aa353b"} Oct 30 19:04:34.406348 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.406315 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5m5f" event={"ID":"53bc3361-f8d3-49de-8490-baf98084ac36","Type":"ContainerStarted","Data":"ef628d54ce6be9b5dc80e1e296c11e5901c16ff6888d38ef62ae5cc59b44330d"} Oct 30 19:04:34.406719 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.406371 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z5m5f" event={"ID":"53bc3361-f8d3-49de-8490-baf98084ac36","Type":"ContainerStarted","Data":"cb0968f11a987ffb3cb1b915494f299a15a9d676f7199db4c389d3e9385d49ae"} Oct 30 19:04:34.465837 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.465788 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z5m5f" podStartSLOduration=2.464507012 podStartE2EDuration="3.465774711s" totalImagesPullingTime="934.537221ms" totalInitContainerRuntime="66.730478ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.46515354 +0000 UTC m=+52.889689354" watchObservedRunningTime="2026-10-30 19:04:34.465774711 +0000 UTC m=+52.890310581" Oct 30 19:04:34.502532 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.502502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.640273 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.640234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.767455 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.767361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.921392 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:34.921357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:35.051657 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:35.051633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:35.051784 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:35.051660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:35.051784 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:35.051727 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:35.051849 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:35.051784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:35.051880 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:35.051852 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:35.051937 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:35.051910 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:35.577823 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:35.577776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:36.079854 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:36.079824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:36.220828 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:36.220795 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:37.030178 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.030149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:37.052233 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.052212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:37.052345 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.052214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:37.052389 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.052214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:37.052422 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:37.052389 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:37.052464 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:37.052302 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:37.052502 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:37.052458 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:37.168311 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.168284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:37.526448 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.526417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:37.682416 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:37.682389 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:38.303373 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:38.303339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:38.467367 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:38.467337 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:38.607778 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:38.607658 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:39.052218 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.052197 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:39.052218 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.052215 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:39.052387 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.052233 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:39.052387 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:39.052307 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhhzq" podUID="e8b22f35-75f3-4149-a2aa-5ee5074fa474" Oct 30 19:04:39.052387 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:39.052361 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rzn" podUID="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" Oct 30 19:04:39.052481 ip-10-0-135-68 kubenswrapper[2582]: E1030 19:04:39.052416 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9kwwp" podUID="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" Oct 30 19:04:39.825132 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.825109 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeReady" Oct 30 19:04:39.825434 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.825229 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:04:39.880588 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.880562 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lxr6q"] Oct 30 19:04:39.882302 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.882289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:39.886678 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.886656 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fxcl2"] Oct 30 19:04:39.888345 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.888323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:39.898928 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.898909 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:04:39.899008 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.898927 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9w5x\"" Oct 30 19:04:39.899048 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.899008 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:04:39.904152 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.904134 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lxr6q"] Oct 30 19:04:39.908616 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.908595 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxcl2"] Oct 30 19:04:39.961393 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:39.961515 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:39.961562 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:39.961562 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98tv\" (UniqueName: \"kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:39.961632 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:39.961632 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.961597 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rtt\" (UniqueName: \"kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:39.984955 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.984930 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dw5vv"] Oct 30 19:04:39.988178 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.988165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:39.999308 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.999288 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bwrqq\"" Oct 30 19:04:39.999397 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.999312 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:04:39.999397 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.999384 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:04:39.999475 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:39.999409 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:04:40.007403 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.007384 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:04:40.008585 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.008564 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dw5vv"] Oct 30 19:04:40.062697 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.062789 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.062789 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98tv\" (UniqueName: \"kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.062860 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062823 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.062952 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rtt\" (UniqueName: \"kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:40.062995 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.063045 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.062994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:40.063045 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.063173 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.063295 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.063359 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.063359 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063350 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.063538 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.063523 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.065220 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.065202 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.065302 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.065280 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:40.083546 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.083525 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98tv\" (UniqueName: \"kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.083762 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.083741 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rtt\" (UniqueName: \"kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:40.099832 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.099809 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:04:40.099923 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.099909 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:04:40.099973 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.099931 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:04:40.100005 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.099977 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbgr5\"" Oct 30 19:04:40.164275 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164367 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164367 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164361 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164451 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164491 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164491 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164772 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164752 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.164973 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.164954 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.166247 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.166229 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.184933 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.184912 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.193412 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.193398 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:40.198863 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.198844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:40.302970 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.302945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:40.499379 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.499350 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxcl2"] Oct 30 19:04:40.502278 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:40.502252 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e563a40_316b_454d_8e73_fa7d8fd091b3.slice/crio-15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23 WatchSource:0}: Error finding container 15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23: Status 404 returned error can't find the container with id 15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23 Oct 30 19:04:40.512193 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.512170 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lxr6q"] Oct 30 19:04:40.514594 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:40.514573 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9712bd0c_c603_4d44_a99b_32bd83dc414c.slice/crio-feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780 WatchSource:0}: Error finding container feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780: Status 404 returned error can't find the container with id feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780 Oct 30 19:04:40.595846 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.595786 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dw5vv"] Oct 30 19:04:40.598343 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:40.598321 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae25ecb1_8d46_432e_9ef8_dcd2dd38bcab.slice/crio-b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26 WatchSource:0}: Error finding container b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26: Status 404 returned error can't find the container with id b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26 Oct 30 19:04:40.747364 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:40.747338 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:41.052255 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.052223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:41.052255 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.052246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:41.052829 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.052232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:41.068846 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.068819 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:04:41.068972 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.068917 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:04:41.069561 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.069543 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:04:41.069677 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.069547 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\"" Oct 30 19:04:41.069677 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.069597 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\"" Oct 30 19:04:41.069810 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.069697 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\"" Oct 30 19:04:41.069810 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.069741 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:04:41.423728 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.423693 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dw5vv" event={"ID":"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab","Type":"ContainerStarted","Data":"a01e48fe42d8e43c9d460da00771c9d6c625605150e59a5ba0eba947c25b054d"} Oct 30 19:04:41.423847 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.423781 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dw5vv" event={"ID":"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab","Type":"ContainerStarted","Data":"0eaf6e67f01310e59e8dca004e538790fd6784385bf49bb6de00809f715be732"} Oct 30 19:04:41.423847 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.423812 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dw5vv" event={"ID":"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab","Type":"ContainerStarted","Data":"b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26"} Oct 30 19:04:41.425207 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.425180 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxr6q" event={"ID":"9712bd0c-c603-4d44-a99b-32bd83dc414c","Type":"ContainerStarted","Data":"feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780"} Oct 30 19:04:41.426694 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:41.426671 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxcl2" event={"ID":"4e563a40-316b-454d-8e73-fa7d8fd091b3","Type":"ContainerStarted","Data":"15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23"} Oct 30 19:04:42.434215 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:42.434175 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxr6q" event={"ID":"9712bd0c-c603-4d44-a99b-32bd83dc414c","Type":"ContainerStarted","Data":"097efb94e2b75ce123fc01985997044bbb8e5c09fb678573292cd7c1bdf48322"} Oct 30 19:04:43.390470 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.390438 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:43.438073 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.437985 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxcl2" event={"ID":"4e563a40-316b-454d-8e73-fa7d8fd091b3","Type":"ContainerStarted","Data":"93cb803049e8c4a26ed6d972125b74885d015c5387ed5db3f841d557dc4ab164"} Oct 30 19:04:43.440007 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.439985 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dw5vv" event={"ID":"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab","Type":"ContainerStarted","Data":"eddd523c822b74c2def6a4ab2c88c7fbd545fca728eed95cfe6b88bd35f6b77b"} Oct 30 19:04:43.441662 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.441643 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxr6q" event={"ID":"9712bd0c-c603-4d44-a99b-32bd83dc414c","Type":"ContainerStarted","Data":"501df1379e7ab091f2eeff6f72fb96febdd626705852a497c7dd13e94e583a42"} Oct 30 19:04:43.441732 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.441723 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:43.482103 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.482052 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fxcl2" podStartSLOduration=1.797725199 podStartE2EDuration="4.482040142s" totalImagesPullingTime="2.684314943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:43.482035007 +0000 UTC m=+61.906570821" watchObservedRunningTime="2026-10-30 19:04:43.482040142 +0000 UTC m=+61.906575953" Oct 30 19:04:43.529313 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.529138 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lxr6q" podStartSLOduration=2.739022054 podStartE2EDuration="4.529125262s" totalImagesPullingTime="1.790103208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:43.528295746 +0000 UTC m=+61.952831561" watchObservedRunningTime="2026-10-30 19:04:43.529125262 +0000 UTC m=+61.953661075" Oct 30 19:04:43.544401 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.544365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:43.591037 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:43.590992 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dw5vv" podStartSLOduration=2.450767582 podStartE2EDuration="4.590979194s" totalImagesPullingTime="2.140211612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:04:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:43.589575646 +0000 UTC m=+62.014111461" watchObservedRunningTime="2026-10-30 19:04:43.590979194 +0000 UTC m=+62.015515008" Oct 30 19:04:44.233254 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:44.233210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:44.404829 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:44.404797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.403856 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.403824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.548434 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.548402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.683794 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.683714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.815387 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.815360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:46.815540 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.815393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:46.831213 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.831187 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:04:46.838173 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.838147 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:46.845553 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.845533 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:04:46.857567 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.857546 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:46.916534 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.916512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:46.932226 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.932203 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:04:46.952838 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.952789 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:04:46.960240 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:46.960217 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:47.083048 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.083021 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\"" Oct 30 19:04:47.088146 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.088125 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:47.106229 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.106209 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\"" Oct 30 19:04:47.110233 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.110209 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:47.111229 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.111213 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\"" Oct 30 19:04:47.115924 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.115906 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:47.410302 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.410123 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhhzq"] Oct 30 19:04:47.412866 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:47.412836 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b22f35_75f3_4149_a2aa_5ee5074fa474.slice/crio-78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632 WatchSource:0}: Error finding container 78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632: Status 404 returned error can't find the container with id 78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632 Oct 30 19:04:47.437375 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.437349 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9rzn"] Oct 30 19:04:47.439799 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:47.439768 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded069dac_23ef_4ec7_88a9_e74dc4e4161b.slice/crio-1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7 WatchSource:0}: Error finding container 1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7: Status 404 returned error can't find the container with id 1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7 Oct 30 19:04:47.445447 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.445364 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9kwwp"] Oct 30 19:04:47.447544 ip-10-0-135-68 kubenswrapper[2582]: W1030 19:04:47.447524 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2715ff_fc4b_46c6_89fe_f95275cfe46d.slice/crio-0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49 WatchSource:0}: Error finding container 0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49: Status 404 returned error can't find the container with id 0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49 Oct 30 19:04:47.453915 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.453891 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rzn" event={"ID":"ed069dac-23ef-4ec7-88a9-e74dc4e4161b","Type":"ContainerStarted","Data":"1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7"} Oct 30 19:04:47.455121 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.455079 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9kwwp" event={"ID":"fc2715ff-fc4b-46c6-89fe-f95275cfe46d","Type":"ContainerStarted","Data":"0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49"} Oct 30 19:04:47.456192 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.456171 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhhzq" event={"ID":"e8b22f35-75f3-4149-a2aa-5ee5074fa474","Type":"ContainerStarted","Data":"78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632"} Oct 30 19:04:47.841430 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.841393 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.987812 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:47.987771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:49.465406 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:49.465370 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rzn" event={"ID":"ed069dac-23ef-4ec7-88a9-e74dc4e4161b","Type":"ContainerStarted","Data":"62eb5b5b801540b698c9c24a0aed1d8596d7e2f0d09babb175f418999326f76d"} Oct 30 19:04:49.465864 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:49.465431 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rzn" event={"ID":"ed069dac-23ef-4ec7-88a9-e74dc4e4161b","Type":"ContainerStarted","Data":"24a6dd16453185b04d849be496e25ae5dbb4e8217421b952069bcdd381da523e"} Oct 30 19:04:49.507718 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:49.507650 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l9rzn" podStartSLOduration=66.473551966 podStartE2EDuration="1m7.507632843s" totalImagesPullingTime="1.034080877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:49.506977268 +0000 UTC m=+67.931513097" watchObservedRunningTime="2026-10-30 19:04:49.507632843 +0000 UTC m=+67.932168661" Oct 30 19:04:50.351685 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:50.351652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.487613 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:50.487578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:51.474638 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:51.474605 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhhzq" event={"ID":"e8b22f35-75f3-4149-a2aa-5ee5074fa474","Type":"ContainerStarted","Data":"e595fde434254cd68dac2f35b1b16753311476956001c5d31b2e9413bbab6177"} Oct 30 19:04:51.474917 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:51.474769 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:51.517667 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:51.517614 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nhhzq" podStartSLOduration=66.363177559 podStartE2EDuration="1m9.517600356s" totalImagesPullingTime="3.154422797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:51.516767551 +0000 UTC m=+69.941303368" watchObservedRunningTime="2026-10-30 19:04:51.517600356 +0000 UTC m=+69.942136169" Oct 30 19:04:52.204902 ip-10-0-135-68 kubenswrapper[2582]: I1030 19:04:52.204800 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:52.204970 ip-10-0-135-68 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:04:52.227548 ip-10-0-135-68 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:04:52.227817 ip-10-0-135-68 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:04:52.227883 ip-10-0-135-68 systemd[1]: kubelet.service: Consumed 3.551s CPU time. Oct 30 19:04:52.250767 ip-10-0-135-68 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:04:52.319419 ip-10-0-135-68 kubenswrapper[4990]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:52.319419 ip-10-0-135-68 kubenswrapper[4990]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:04:52.319419 ip-10-0-135-68 kubenswrapper[4990]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:52.319419 ip-10-0-135-68 kubenswrapper[4990]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:52.321470 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321394 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.321470 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321470 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.321470 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321474 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321477 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321481 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321491 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321494 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321497 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321500 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321502 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321505 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321508 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321510 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321513 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321516 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321518 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321521 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321524 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321527 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321530 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321532 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.321563 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321535 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321537 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321540 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321542 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321545 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321548 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321551 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321554 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321557 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321559 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321562 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321565 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321567 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321570 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321573 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321576 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321579 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321582 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321584 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321587 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.322040 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321589 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321593 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321598 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321601 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321604 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321606 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321609 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321611 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321614 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321617 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321620 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321623 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321625 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321628 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321632 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321636 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321640 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321643 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321647 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.322601 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321650 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321653 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321656 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321659 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321662 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321665 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321668 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321671 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321674 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321677 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321682 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321685 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321687 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321690 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321693 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321695 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321698 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321700 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321703 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321705 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.323198 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321708 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321710 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321713 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321716 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321719 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321721 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321723 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321726 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321729 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321732 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321735 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321737 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321740 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321743 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321745 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321748 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321751 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321753 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321756 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.323708 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321758 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321761 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321763 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321766 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321770 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321772 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.321775 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322542 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322549 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322552 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322555 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322557 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322560 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322563 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322565 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322568 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322571 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322574 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322576 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322579 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.324203 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322582 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322584 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322587 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322590 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322593 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322596 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322599 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322601 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322604 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322607 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322610 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322613 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322615 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322618 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322620 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322623 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322627 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322630 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322632 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322635 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.324755 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322638 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322641 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322643 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322646 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322649 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322652 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322654 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322657 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322660 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322662 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322665 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322667 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322670 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322673 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322676 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322679 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322681 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322684 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322686 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322689 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.325263 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322692 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322694 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322697 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322699 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322702 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322705 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322707 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322710 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322712 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322715 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322720 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322725 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322731 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322734 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322737 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322739 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322747 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322750 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322754 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.325813 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322757 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322760 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322762 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322765 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322768 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322771 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322774 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322777 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322780 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322782 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322785 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322787 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322790 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322793 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322795 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322798 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322802 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322804 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322807 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322810 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.326306 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322812 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322815 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322817 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322820 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322823 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322827 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322830 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322833 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322835 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322838 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322840 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322843 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322845 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.322848 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323325 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323331 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323334 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323337 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323340 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.326802 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323342 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323345 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323348 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323351 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323354 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323356 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323359 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323361 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323364 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323367 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323369 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323372 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323375 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323377 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323380 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323382 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323385 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323388 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323393 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323395 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.327285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323398 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323401 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323403 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323406 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323409 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323411 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323414 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323416 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323419 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323421 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323425 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323427 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323430 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323432 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323435 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323437 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323440 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323443 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323445 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323447 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.327944 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323450 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323452 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323455 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323457 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323460 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323462 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323465 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323468 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323470 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323473 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323483 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323486 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323489 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323492 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323495 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323497 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323500 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323502 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323505 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.328804 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323507 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323510 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323513 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323515 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323518 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323521 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323525 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323528 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323531 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323533 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323536 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323539 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323541 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323543 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323546 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323548 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323551 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323553 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323556 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.329582 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323558 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323561 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323565 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323568 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323572 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323580 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323583 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323586 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323588 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323591 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323594 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323598 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323602 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323605 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323608 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323610 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323613 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323616 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323618 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.330517 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323621 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323624 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323627 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.323629 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323722 4990 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323739 4990 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323748 4990 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323754 4990 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323760 4990 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323763 4990 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323768 4990 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323772 4990 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323775 4990 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323787 4990 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323790 4990 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323797 4990 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323801 4990 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323804 4990 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323807 4990 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323811 4990 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323814 4990 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323817 4990 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323822 4990 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323825 4990 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:04:52.331405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323828 4990 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323831 4990 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323835 4990 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323839 4990 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323842 4990 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323845 4990 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323849 4990 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323852 4990 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323855 4990 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323858 4990 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323861 4990 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323864 4990 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323868 4990 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323871 4990 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323874 4990 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323877 4990 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323881 4990 flags.go:64] FLAG: --enable-server="true" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323883 4990 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323887 4990 flags.go:64] FLAG: --event-burst="100" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323890 4990 flags.go:64] FLAG: --event-qps="50" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323893 4990 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323897 4990 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323900 4990 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323905 4990 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:04:52.332515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323908 4990 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323912 4990 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323915 4990 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323919 4990 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323922 4990 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323926 4990 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323929 4990 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323932 4990 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323935 4990 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323938 4990 flags.go:64] FLAG: --feature-gates="" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323942 4990 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323945 4990 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323949 4990 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323952 4990 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323955 4990 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323958 4990 flags.go:64] FLAG: --help="false" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323962 4990 flags.go:64] FLAG: --hostname-override="ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323965 4990 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323968 4990 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323971 4990 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323974 4990 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323978 4990 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323981 4990 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:04:52.333628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323984 4990 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323987 4990 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323990 4990 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323993 4990 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323996 4990 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.323999 4990 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324003 4990 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324005 4990 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324010 4990 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324013 4990 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324016 4990 flags.go:64] FLAG: --lock-file="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324019 4990 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324022 4990 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324025 4990 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324031 4990 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324035 4990 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324038 4990 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324041 4990 flags.go:64] FLAG: --logging-format="text" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324044 4990 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324047 4990 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324050 4990 flags.go:64] FLAG: --manifest-url="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324053 4990 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324057 4990 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324061 4990 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324064 4990 flags.go:64] FLAG: --max-pods="110" Oct 30 19:04:52.334737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324067 4990 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324071 4990 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324074 4990 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324077 4990 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324080 4990 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324083 4990 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324100 4990 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324107 4990 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324111 4990 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324114 4990 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324117 4990 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324120 4990 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324123 4990 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324126 4990 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324129 4990 flags.go:64] FLAG: --port="10250" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324133 4990 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324137 4990 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040809bb5903bb00a" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324140 4990 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324143 4990 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324146 4990 flags.go:64] FLAG: --register-node="true" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324149 4990 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324153 4990 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324157 4990 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324160 4990 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324163 4990 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324167 4990 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:04:52.335834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324170 4990 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324173 4990 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324176 4990 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324179 4990 flags.go:64] FLAG: --runonce="false" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324183 4990 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324186 4990 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324189 4990 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324193 4990 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324197 4990 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324200 4990 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324203 4990 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324206 4990 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324209 4990 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324213 4990 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324216 4990 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324219 4990 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324222 4990 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324225 4990 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324228 4990 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324233 4990 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324236 4990 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324240 4990 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324244 4990 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324247 4990 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324250 4990 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:04:52.336987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324253 4990 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324256 4990 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324259 4990 flags.go:64] FLAG: --v="2" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324269 4990 flags.go:64] FLAG: --version="false" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324274 4990 flags.go:64] FLAG: --vmodule="" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324278 4990 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324281 4990 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324415 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324419 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324422 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324424 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324427 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324430 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324433 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324437 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324440 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324443 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324446 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324448 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324451 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324454 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.337922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324456 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324459 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324462 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324464 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324467 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324469 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324472 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324477 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324480 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324482 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324485 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324487 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324490 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324493 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324495 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324498 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324501 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324504 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324506 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324509 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.338644 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324512 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324515 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324519 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324523 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324527 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324532 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324535 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324538 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324540 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324543 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324546 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324548 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324551 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324554 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324557 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324560 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324562 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324565 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.339174 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324567 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324571 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324574 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324576 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324579 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324581 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324584 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324587 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324589 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324592 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324595 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324598 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324600 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324603 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324606 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324608 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324611 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324614 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324616 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324620 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.339651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324623 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324626 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324628 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324631 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324634 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324636 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324639 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324641 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324644 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324647 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324649 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324652 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324655 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324658 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324661 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324664 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324667 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324669 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324672 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324675 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.340322 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324677 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324680 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324683 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324685 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324688 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324691 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324694 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324696 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324699 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324701 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324704 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324708 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324711 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324714 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.324739 4990 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324773 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.340865 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324778 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324781 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324784 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324786 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324789 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324792 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324794 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324797 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324800 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324802 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324806 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324809 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324812 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324814 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324817 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324820 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324823 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324825 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324828 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.341383 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324831 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324833 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324836 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324839 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324842 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324845 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324847 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324850 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324854 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324856 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324859 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324861 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324864 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324867 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324869 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324872 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324875 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324877 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324880 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.341879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324882 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324885 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324888 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324890 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324895 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324897 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324900 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324903 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324905 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324908 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324911 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324913 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324916 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324919 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324921 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324924 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324927 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324930 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324933 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324936 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.342419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324938 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324942 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324945 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324948 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324950 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324953 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324955 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324958 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324961 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324965 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324968 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324973 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324977 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324981 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324984 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324987 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324991 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324994 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.324997 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.342906 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325001 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325004 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325006 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325009 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325011 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325014 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325017 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325020 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325022 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325025 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325028 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325031 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325033 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325036 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325040 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325043 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325045 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325048 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325050 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325053 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.343429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325055 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325058 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325060 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325063 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325066 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325068 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325071 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.325074 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.330335 4990 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.330356 4990 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.330377 4990 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.330385 4990 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330474 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330482 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330487 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330493 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330498 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330502 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330507 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.343936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330511 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330516 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330520 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330524 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330529 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330533 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330537 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330547 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330552 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330557 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330562 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330566 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330571 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330576 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330580 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330585 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330589 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330593 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330598 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330602 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.344494 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330607 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330611 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330618 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330623 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330628 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330634 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330638 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330643 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330648 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330652 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330657 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330661 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330665 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330670 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330674 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330678 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330683 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330687 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330691 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.345165 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330697 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330702 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330707 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330711 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330715 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330720 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330724 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330728 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330733 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330737 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330743 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330747 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330752 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330756 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330761 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330765 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330769 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330774 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330778 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330783 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.345854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330787 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330791 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330796 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330800 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330804 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330809 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330813 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330817 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330823 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330828 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330832 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330836 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330841 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330849 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330856 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330861 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330866 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330870 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330875 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.346670 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330879 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330884 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330888 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330892 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330897 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330902 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330906 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330910 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330914 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330919 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330924 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330928 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330932 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330936 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330941 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330945 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330949 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330954 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330959 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.347273 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330963 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.330967 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.331010 4990 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331247 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331257 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331261 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331266 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331271 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331278 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331282 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331287 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331291 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331296 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331300 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331305 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331310 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.347870 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331316 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331320 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331324 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331329 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331333 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331337 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331341 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331346 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331350 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331354 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331358 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331363 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331367 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331372 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331377 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331381 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331386 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331390 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331395 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331399 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.348288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331403 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331407 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331411 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331416 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331420 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331424 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331429 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331433 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331437 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331441 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331445 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331450 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331454 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331459 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331463 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331467 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331471 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331476 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331480 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.348784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331484 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331489 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331493 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331497 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331502 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331506 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331510 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331515 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331519 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331523 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331528 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331539 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331544 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331548 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331552 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331556 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331561 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331565 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331569 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.349283 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331578 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331585 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331590 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331595 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331599 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331604 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331608 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331612 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331616 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331621 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331625 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331630 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331634 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331638 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331642 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331647 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331651 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331655 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331660 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.349762 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331666 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331672 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331678 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331684 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331688 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331694 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331698 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331702 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331707 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331711 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331716 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331721 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331725 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331729 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331735 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331740 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.350390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.331794 4990 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331847 4990 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331853 4990 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331858 4990 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331862 4990 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331869 4990 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331876 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331880 4990 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331885 4990 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331889 4990 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331893 4990 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331898 4990 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331902 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331906 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331911 4990 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331915 4990 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331919 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331923 4990 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331928 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:52.350871 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331932 4990 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331937 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331944 4990 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331948 4990 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331953 4990 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331959 4990 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331965 4990 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331969 4990 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331974 4990 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331979 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331983 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331987 4990 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331992 4990 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.331997 4990 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332001 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332006 4990 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332010 4990 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332015 4990 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332019 4990 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:52.351349 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332023 4990 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332028 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332032 4990 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332036 4990 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332045 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332049 4990 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332054 4990 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332059 4990 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332063 4990 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332068 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332072 4990 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332077 4990 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332082 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332101 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332106 4990 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332113 4990 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332117 4990 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332123 4990 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332127 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332131 4990 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:52.351807 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332135 4990 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332139 4990 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332144 4990 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332148 4990 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332152 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332157 4990 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332163 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332167 4990 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332173 4990 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332177 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332182 4990 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332187 4990 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332191 4990 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332195 4990 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332199 4990 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332203 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332210 4990 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332215 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332220 4990 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332224 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:52.352294 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332228 4990 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332233 4990 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332237 4990 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332241 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332245 4990 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332250 4990 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332254 4990 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332260 4990 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332265 4990 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332269 4990 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332274 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332278 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332283 4990 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332287 4990 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332294 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332298 4990 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332302 4990 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332307 4990 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332311 4990 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:52.352836 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332317 4990 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332322 4990 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332326 4990 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332330 4990 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332335 4990 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332339 4990 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332343 4990 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332347 4990 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332352 4990 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:04:52.332358 4990 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.332435 4990 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.333992 4990 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.334057 4990 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.334333 4990 server.go:1014] "Starting client certificate rotation" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.334467 4990 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.334509 4990 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:41 +0000 UTC" deadline="2028-05-18 00:52:30.155148362 +0000 UTC" Oct 30 19:04:52.353319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.334534 4990 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13565h47m37.820615987s" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.335032 4990 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.336672 4990 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.337992 4990 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.339577 4990 log.go:25] "Validated CRI v1 image API" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.340317 4990 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:04:52.353735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.341869 4990 fs.go:135] Filesystem UUIDs: map[38cf0db0-3fff-4b11-b529-8bb4ec7296cf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80095121-85be-4501-b4cb-1ab6d0e95113:/dev/nvme0n1p4] Oct 30 19:04:52.353899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.341887 4990 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/079acf535272fe24b9ebd036cc8c3d186aeeb53bab85003d7929437784c751f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/079acf535272fe24b9ebd036cc8c3d186aeeb53bab85003d7929437784c751f4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~projected/kube-api-access-x5rtt:{mountpoint:/var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~projected/kube-api-access-x5rtt major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~projected/kube-api-access-zplfk:{mountpoint:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~projected/kube-api-access-zplfk major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~projected/kube-api-access-9jc45:{mountpoint:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~projected/kube-api-access-9jc45 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d5e6381-5d5c-4d33-86cb-3e528c197bcd/volumes/kubernetes.io~projected/kube-api-access-chshm:{mountpoint:/var/lib/kubelet/pods/5d5e6381-5d5c-4d33-86cb-3e528c197bcd/volumes/kubernetes.io~projected/kube-api-access-chshm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61da23fa-4dc5-4973-af62-c338ecdff26d/volumes/kubernetes.io~projected/kube-api-access-6rnc4:{mountpoint:/var/lib/kubelet/pods/61da23fa-4dc5-4973-af62-c338ecdff26d/volumes/kubernetes.io~projected/kube-api-access-6rnc4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67705ac6-bf6a-4e9e-9749-691459286cde/volumes/kubernetes.io~projected/kube-api-access-bn5nk:{mountpoint:/var/lib/kubelet/pods/67705ac6-bf6a-4e9e-9749-691459286cde/volumes/kubernetes.io~projected/kube-api-access-bn5nk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~projected/kube-api-access-v998c:{mountpoint:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~projected/kube-api-access-v998c major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/volumes/kubernetes.io~projected/kube-api-access-fz2hs:{mountpoint:/var/lib/kubelet/pods/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/volumes/kubernetes.io~projected/kube-api-access-fz2hs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~projected/kube-api-access-t98tv:{mountpoint:/var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~projected/kube-api-access-t98tv major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~projected/kube-api-access-wwsxw:{mountpoint:/var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~projected/kube-api-access-wwsxw major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd2c4560-125c-4c15-a15e-9e93a59e9a06/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cd2c4560-125c-4c15-a15e-9e93a59e9a06/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e066ed65-096a-4d83-bcf0-cc3cb10267fc/volumes/kubernetes.io~projected/kube-api-access-frxq5:{mountpoint:/var/lib/kubelet/pods/e066ed65-096a-4d83-bcf0-cc3cb10267fc/volumes/kubernetes.io~projected/kube-api-access-frxq5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/volumes/kubernetes.io~projected/kube-api-access-jsrgr:{mountpoint:/var/lib/kubelet/pods/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/volumes/kubernetes.io~projected/kube-api-access-jsrgr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b22f35-75f3-4149-a2aa-5ee5074fa474/volumes/kubernetes.io~projected/kube-api-access-zrgs6:{mountpoint:/var/lib/kubelet/pods/e8b22f35-75f3-4149-a2aa-5ee5074fa474/volumes/kubernetes.io~projected/kube-api-access-zrgs6 major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~projected/kube-api-access-vq695:{mountpoint:/var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~projected/kube-api-access-vq695 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc2715ff-fc4b-46c6-89fe-f95275cfe46d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fc2715ff-fc4b-46c6-89fe-f95275cfe46d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e4354d5dbcb25f5d11e61cd49141243e704d8f2823574848f7ea769e3e4ce431/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e4751ca2a2596058f4e0f8238c6caba7e2e6d0a09cbf1cf944ea5e242a78263a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6cd41f1347f5cfe03ed58c8348988d5d77b81892e24681e6d8c514f53c101fdc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d928e5f3650eb8eb6ddd5cca55159d009502f6e4c854c5818929303f8a832bc9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2d8c407ccaeaf50f726a5672f5c25f8707ae86c4d99fccc9fa091abbdb3b6664/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d9ee821d6998774f09e1d3f9c836e821ccbb5828fa6361b2c16a031f9a454933/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a5ffd0e4b34eb5510c0a395b1e6eeeefc4c944fc15c3a5743aa34f384b371e66/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3b382283c1163218835c55fac7b55974d83a0da775a2febddc9b77cd2a9f8fd9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e1dc346561724b50b7f6856925701b2d5a7347b6873fc38d052a98a81173ef22/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/6ea4e068156aa8aa9e9891d3eff5b7105d2c5ef25a8c7bfcd151e158a4129b41/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/1b41877fa0c1e76b69b243fa506240fd69a63d68401c2b3baea49ee0e02bbbb0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/aa5772d33fc3b184f51a28baae318713bda893dc91f15eab05c0f409d1af55b5/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3d86c7772f18c7dbcada483d5165dbbb6a45769d03bb8230fd9eaa1d1e988162/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8fbbb8564d709163ff5a84a655b678bb518d71d84f415138642008ec79c82f56/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0b23ca3c3802f92456345c5f983d9dfdbb18f8e2c06501161bafc1fdbe8e9301/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9dbe0282b9d35e9bf4a698eca8097e7c4665fefa40126ec315e2cb66241528a4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/162b7863f2679ce00e9b4edbf5ce3f08a89c3e79c2b942559ba0364dfc7c0fc8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/d6769df92fddcdb8a61f2440dc2197157d5260a47cc9616469a3d771de8e1cb9/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/5eb67da897c1cc7fad3df19da65aad9ea051882bbbf4d463797ddd2969197cc9/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6109dfc085ef2a03dcb94a6df17883845cadddedb2b769ba2fad88cd369fe4b0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/98427abd0cbf6cf1a07ca2c673310ad91955fcb36ee957f81002ab500a851eed/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/1f7af46824eac068126d0d895d9d0519090bbcb70dc2f29d77d03040268489bc/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/b5228dd1c55d4aee61a18b587458b35b470c0281a214469b51232144713ba8e9/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/7d4835148ce322baa8180f820fe6a2c6b0cdcbbba23798a60e0c2ec417499091/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/a56e825431365fe8f7b0a2d7924fbb58d6c093500ea12db0be69ebe1651a273e/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0b5dd005a050b95d92d29cef43fec3f3eaffd3e5d7b7838f271f693b1c957189/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/3d6e3eaffcbe40d599416805a38bb79aa1ee9cb6e45dce95d5f102e3eb81c489/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/ec31ee14b030f4c26ed57ccb8ed5b2cf4eeddb52709976fe22153d326887bdd3/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/89f9fd16b045831605d3e7a5d6d09a1eb4ea8c0b355b3edd3063b30f2151caf9/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/996d306976b4729e08b59f5a8a03f328d28a4c22090a9df92990732550ff08e4/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/f616f4e3a6bfcdc917b4980b4f111ba1c12a6f5b3e99716181035c6608a00443/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/19da77f404e67664fafd9521f1140fbf31465e853ac65d67090d402d94d5dbd8/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/29498b201ba065806d232fa7471070f5bd84c95416939d212a3e396a6270f357/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b4cc2395774dee17379934a861bc5ec6991da0b3d13f38002f928e83d77f71cf/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/98c8b5a35134f1ce29e7f264a3fab9ab231e7b7ece9521f2941e8c6172f0ac1f/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/9eb72bd2fc6f23ca6a93c5af57572c7bbcd9b0d6fb27beacbe8c6964e1f677ee/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/6ccaec4b9a8cc2e76228e7e24e3bf4812c7f03637317b2c57cda27de1dafbb12/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/18abb0fd884e0c955a9355df581e3f826bc476489c241c1640adc4b481c4341a/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/63ae2e95feffbe83be11aa90316ae1fd666b1d8bf97c79ec17638677b97eec08/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/db6d564c70c8ec071b0cce19a53e93bcc1666077c0981af8331609575b67a71a/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f8b7df0083f452561277d5e0fcff57358248269a5e3a40664fd7eb8f3d8f3788/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0e4aaa29d54106ec256a725284296406014c3005e46e82a4ae12760016938ee0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e76ddb2607d97a8d80ddebd22f3c773308a6d51dd642ab43496e39ffa234fc05/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/739ea07315215c063fed707e304e8ddd8dab869ca5c550d4594299c160c6697d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/ecbbfd6a5749708c28d6ed6506b2efcbb7bf9e465b2688e05bd0ec6360dee5d8/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/1d4d197ca3eee060bbf2f8a4b1bfb1b6ec3b108561bca5faa81ba4311d11b478/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/45aedb47500baa9866d85cc606dca734854a76f86f71c135350a51799edb97ef/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e610a9c57d84308eef53ab035e9433a5286346a8f2be28376bb9c50a66d68608/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f07049b0aeaa4bd426d85a1c623364641f8709b83bfbdd69baefad860a24bf57/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6ea538229be3cbdc853a8e90b734220b9043c7b9c62bac0dc2baa9e50a1ccf66/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.348731 4990 manager.go:217] Machine: {Timestamp:2026-10-30 19:04:52.348283717 +0000 UTC m=+0.059809075 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103867 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27aaebdede8c2d37b6b579b69eb0bf SystemUUID:ec27aaeb-dede-8c2d-37b6-b579b69eb0bf BootID:5ba282a1-fa59-4efe-a293-af1affb5762d Filesystems:[{Device:/var/lib/kubelet/pods/e8b22f35-75f3-4149-a2aa-5ee5074fa474/volumes/kubernetes.io~projected/kube-api-access-zrgs6 DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~projected/kube-api-access-v998c DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56560ba07bea61765f52a8ebd9e46ff44d8f4b5471c8ea318c9f8b7d70e2cdcb/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e066ed65-096a-4d83-bcf0-cc3cb10267fc/volumes/kubernetes.io~projected/kube-api-access-frxq5 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d49a339cf050ed1de2019d3471cd018dbce4ff4fe14687fc3dec29c34e19084a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~projected/kube-api-access-x5rtt DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1295a76759e343ea519de5e97ca522b6f910a05ee7cdac14210e8319217533a7/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~projected/kube-api-access-zplfk DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/volumes/kubernetes.io~projected/kube-api-access-jsrgr DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~projected/kube-api-access-t98tv DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~projected/kube-api-access-wwsxw DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~projected/kube-api-access-vq695 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fd9e114-e938-4928-94e8-c9c3bd7ae37b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61da23fa-4dc5-4973-af62-c338ecdff26d/volumes/kubernetes.io~projected/kube-api-access-6rnc4 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feea1a3bf7b8cf474b97330cb97b225e26e4f5ee6aa047c6aefa4fc8bc431780/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e81857e36a7ec73ee8788f68265d2ad21ee0725645683081eaa0c680acf5a75/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb66d28e0757ba43a81cff48771bcb4ed04bb97e5730624db212cbb002d6c05f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc2715ff-fc4b-46c6-89fe-f95275cfe46d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e563a40-316b-454d-8e73-fa7d8fd091b3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed069dac-23ef-4ec7-88a9-e74dc4e4161b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:263 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e81c8457efcd523d0ae0bf379a44332762a3669f03b222c296eed77c6b10e578/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9712bd0c-c603-4d44-a99b-32bd83dc414c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15c1cd5096014da54a2eacf048a0f97965c3d26a270548683a85a4bc82c24c23/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd2c4560-125c-4c15-a15e-9e93a59e9a06/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef107f8-59da-48f3-81dc-df4851b26044/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/volumes/kubernetes.io~projected/kube-api-access-fz2hs DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/329a7597c958a3b9a017330bce5ff7fb24c7318c1e5c229af6c6ef9338f5dc4f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/079acf535272fe24b9ebd036cc8c3d186aeeb53bab85003d7929437784c751f4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6de21e37c56449171adefed6ddfd121bb4b8c84b4da0aac2aad1f403e341d26/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/67705ac6-bf6a-4e9e-9749-691459286cde/volumes/kubernetes.io~projected/kube-api-access-bn5nk DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00a34098a843b3897fea5a08fa1e0a6d4b2a52066eb89d775e185d39e46384e7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~projected/kube-api-access-9jc45 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78f9451824d1890b84033a46eadf1ed1de772eeb282afd4454b3a0366008b632/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0382b81fb2f32d7c20f96b1624a8f57eaa8df1b496105ffb7cd090ad26903e49/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18139ca1953d7f37327f283267430d80b4a86da4693fd9ec347953fe5e6c834f/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5d5e6381-5d5c-4d33-86cb-3e528c197bcd/volumes/kubernetes.io~projected/kube-api-access-chshm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53bc3361-f8d3-49de-8490-baf98084ac36/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0382b81fb2f32d7 MacAddress:0a:a0:08:5a:b2:f3 Speed:10000 Mtu:8901} {Name:1295a76759e343e MacAddress:72:ad:13:76:35:58 Speed:10000 Mtu:8901} {Name:15c1cd5096014da MacAddress:da:d1:18:90:ac:b2 Speed:10000 Mtu:8901} {Name:78f9451824d1890 MacAddress:4a:81:45:13:ed:4f Speed:10000 Mtu:8901} {Name:b6de21e37c56449 MacAddress:ee:17:65:6f:d4:6b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:c4:03:fb Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:20:54:5d:79:47 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:c4:03:fb Speed:0 Mtu:9001} {Name:feea1a3bf7b8cf4 MacAddress:d2:37:52:10:85:80 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:56:82:e2:fb:22:c9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:3e:41:a3:e5:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.348990 4990 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349029 4990 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349271 4990 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349287 4990 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349401 4990 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349408 4990 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349418 4990 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349437 4990 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349508 4990 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349647 4990 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349751 4990 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349783 4990 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349792 4990 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349801 4990 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349810 4990 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.349817 4990 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.350366 4990 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.350375 4990 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.350777 4990 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.352846 4990 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.352855 4990 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.352938 4990 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353125 4990 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353153 4990 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353240 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353252 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353258 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353263 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353268 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353274 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353280 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353285 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353292 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353298 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353306 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353316 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353342 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:04:52.354078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.353348 4990 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:04:52.356534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.356522 4990 server.go:1263] "Started kubelet" Oct 30 19:04:52.356822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.356669 4990 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:04:52.356947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.356623 4990 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:04:52.356947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.356930 4990 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:04:52.357056 ip-10-0-135-68 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:04:52.357297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.357247 4990 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:04:52.357900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.357882 4990 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.359971 4990 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360363 4990 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360410 4990 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:42 +0000 UTC" deadline="2028-07-31 02:34:59.26992201 +0000 UTC" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360444 4990 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15343h30m6.909480964s" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.360505 4990 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-68.ec2.internal\" not found" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360510 4990 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360519 4990 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:04:52.360837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.360526 4990 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:04:52.362860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.362686 4990 factory.go:55] Registering systemd factory Oct 30 19:04:52.362860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.362705 4990 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363674 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" volumeName="kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363712 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" volumeName="kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363726 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e563a40-316b-454d-8e73-fa7d8fd091b3" volumeName="kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363738 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" volumeName="kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363760 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" volumeName="kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363773 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363791 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" volumeName="kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363802 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" volumeName="kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363816 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61da23fa-4dc5-4973-af62-c338ecdff26d" volumeName="kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363827 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67705ac6-bf6a-4e9e-9749-691459286cde" volumeName="kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363838 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d9d8a4b-34d0-47ba-abc1-8e71a55d600c" volumeName="kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363850 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d9d8a4b-34d0-47ba-abc1-8e71a55d600c" volumeName="kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363860 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e563a40-316b-454d-8e73-fa7d8fd091b3" volumeName="kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363871 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363885 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67705ac6-bf6a-4e9e-9749-691459286cde" volumeName="kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363896 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef107f8-59da-48f3-81dc-df4851b26044" volumeName="kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363907 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab" volumeName="kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363920 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab" volumeName="kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363932 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363943 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" volumeName="kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363954 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67705ac6-bf6a-4e9e-9749-691459286cde" volumeName="kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363964 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712bd0c-c603-4d44-a99b-32bd83dc414c" volumeName="kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.363987 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712bd0c-c603-4d44-a99b-32bd83dc414c" volumeName="kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364000 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2c4560-125c-4c15-a15e-9e93a59e9a06" volumeName="kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364013 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e066ed65-096a-4d83-bcf0-cc3cb10267fc" volumeName="kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364024 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" volumeName="kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364036 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61da23fa-4dc5-4973-af62-c338ecdff26d" volumeName="kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364048 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5b7c1f1-3b2d-4b73-8870-f1cffe60206e" volumeName="kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364063 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5b7c1f1-3b2d-4b73-8870-f1cffe60206e" volumeName="kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr" seLinuxMountContext="" Oct 30 19:04:52.364119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364075 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed069dac-23ef-4ec7-88a9-e74dc4e4161b" volumeName="kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695" seLinuxMountContext="" Oct 30 19:04:52.365601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.364105 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc2715ff-fc4b-46c6-89fe-f95275cfe46d" volumeName="kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365615 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" volumeName="kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365641 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365652 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b22f35-75f3-4149-a2aa-5ee5074fa474" volumeName="kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365664 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" volumeName="kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365678 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef107f8-59da-48f3-81dc-df4851b26044" volumeName="kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365689 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712bd0c-c603-4d44-a99b-32bd83dc414c" volumeName="kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365701 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls" seLinuxMountContext="" Oct 30 19:04:52.365712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365712 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef107f8-59da-48f3-81dc-df4851b26044" volumeName="kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365725 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9712bd0c-c603-4d44-a99b-32bd83dc414c" volumeName="kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365736 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab" volumeName="kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365747 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab" volumeName="kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365758 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2c4560-125c-4c15-a15e-9e93a59e9a06" volumeName="kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365769 4990 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53bc3361-f8d3-49de-8490-baf98084ac36" volumeName="kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365780 4990 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:04:52.366257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.365788 4990 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:04:52.367044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367018 4990 factory.go:153] Registering CRI-O factory Oct 30 19:04:52.367153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367046 4990 factory.go:223] Registration of the crio container factory successfully Oct 30 19:04:52.367153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367133 4990 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:04:52.367264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367157 4990 factory.go:103] Registering Raw factory Oct 30 19:04:52.367264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367173 4990 manager.go:1199] Started watching for new ooms in manager Oct 30 19:04:52.367506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.367494 4990 manager.go:319] Starting recovery of all containers Oct 30 19:04:52.383161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.383139 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:52.383992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.383967 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:52.390966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.390914 4990 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:04:52.392785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.392762 4990 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:04:52.392785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.392786 4990 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:04:52.392947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.392808 4990 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:04:52.392947 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.392847 4990 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:04:52.399611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.399588 4990 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:04:52.399896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.399882 4990 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:52.401034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.401018 4990 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:52.405115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.404174 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:04:52.405831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.405802 4990 generic.go:356] "Generic (PLEG): container finished" podID="4fd9e114-e938-4928-94e8-c9c3bd7ae37b" containerID="779d85a2a2c767a2aa2b41961fdc28c9ff16aeb1610b3a4a3d3cf86e2c4e1111" exitCode=1 Oct 30 19:04:52.419324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.419306 4990 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:52.420177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.420157 4990 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:52.426601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426578 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="0aa7793bea0325f2d22c7aece8f662b2a6db555d8eb3c3f71788af1e3452ba8d" exitCode=0 Oct 30 19:04:52.426675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426603 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="38493811cf192e9d5ae0f3b3624e54784cdc12a5333dd31f8f331ee2d88c7f68" exitCode=0 Oct 30 19:04:52.426675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426617 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="fb51161ab52b63a158a48570bf72d15351e3d87bfce2efd7b0f2874cb9cacb64" exitCode=0 Oct 30 19:04:52.426675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426629 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="98bc8f17fe31d3558a1d8e280c974c146c592ba309943ab7cfe7df3b887c97e8" exitCode=0 Oct 30 19:04:52.426675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426639 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="24c1dfd33bec0a8b8d84c1a484124e3c205f285b7174c3244a2bc0c77de67f1f" exitCode=0 Oct 30 19:04:52.426675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.426652 4990 generic.go:356] "Generic (PLEG): container finished" podID="5d5e6381-5d5c-4d33-86cb-3e528c197bcd" containerID="7e233076edb60ca7da6ad19e4d929322018faf34644f2bd067a7a2a782c6998e" exitCode=0 Oct 30 19:04:52.433049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.433029 4990 generic.go:356] "Generic (PLEG): container finished" podID="a5e8abc4535060f5a3a432c3ab1497e7" containerID="4c19d7d083650e75f8f19183fe67292a5398806c46907729b52d75935ceae8f7" exitCode=0 Oct 30 19:04:52.437037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.437012 4990 generic.go:356] "Generic (PLEG): container finished" podID="53bc3361-f8d3-49de-8490-baf98084ac36" containerID="f50a31a37ba3d733351e3026bf77a7b46a9e4daef3e56573068d128ce0aa353b" exitCode=0 Oct 30 19:04:52.442784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.442769 4990 manager.go:324] Recovery completed Oct 30 19:04:52.459213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459190 4990 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:04:52.459213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459211 4990 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:04:52.459375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459232 4990 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:04:52.459426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459411 4990 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:04:52.459483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459429 4990 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:04:52.459483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459456 4990 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:04:52.459483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459473 4990 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:04:52.459483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459482 4990 policy_none.go:50] "Start" Oct 30 19:04:52.459673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459496 4990 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:04:52.459673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459509 4990 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:04:52.459766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459745 4990 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:04:52.459766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459758 4990 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:04:52.459863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.459767 4990 policy_none.go:44] "Start" Oct 30 19:04:52.464127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.464086 4990 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:04:52.468667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.468653 4990 manager.go:341] "Starting Device Plugin manager" Oct 30 19:04:52.469117 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.468681 4990 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:04:52.469190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469119 4990 server.go:87] "Starting device plugin registration server" Oct 30 19:04:52.469406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469392 4990 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:04:52.469468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469407 4990 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:04:52.469515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469475 4990 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:04:52.469606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469568 4990 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:04:52.469705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469612 4990 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:04:52.469705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469619 4990 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:04:52.469795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.469669 4990 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:04:52.469601371Z","UUID":"9051d071-a688-4051-9e13-de36b95d3c1e","Handler":null,"Name":"","Endpoint":""} Oct 30 19:04:52.470896 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.470873 4990 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:04:52.471145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.471125 4990 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:04:52.471237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.471173 4990 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:04:52.492984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.492913 4990 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal"] Oct 30 19:04:52.493584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493570 4990 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.493669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493632 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"c1a30bdfa789f3bccffc4563d5bc301a","Type":"ContainerStarted","Data":"8f75cc509fd88386aef1957f9f6549418d05caa0d891ed603d18cdc240b9dfb1"} Oct 30 19:04:52.493708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493686 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" event={"ID":"c1a30bdfa789f3bccffc4563d5bc301a","Type":"ContainerStarted","Data":"ebbcfa556d0454c4b632373b3695f517077b67c3842f4f5cedb818b1bbf7681e"} Oct 30 19:04:52.493742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493708 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerStarted","Data":"b4a41e8be2b2215217bb8210cf176a4dd5a6369ac5095013295f45185c94356c"} Oct 30 19:04:52.493742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493729 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerDied","Data":"4c19d7d083650e75f8f19183fe67292a5398806c46907729b52d75935ceae8f7"} Oct 30 19:04:52.493830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493750 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" event={"ID":"a5e8abc4535060f5a3a432c3ab1497e7","Type":"ContainerStarted","Data":"49ea4d30e319d299d9f6ee5fa49f066c6f9f8e1b182d6088ce624a6488141c37"} Oct 30 19:04:52.493830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.493821 4990 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.516228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.516210 4990 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:52.516296 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.516249 4990 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.516772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.516756 4990 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:52.516848 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:52.516795 4990 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.569736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.569716 4990 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:04:52.571808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.571795 4990 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:04:52.571864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.571823 4990 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:04:52.571864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.571837 4990 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-68.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:04:52.571918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.571880 4990 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.600574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.600556 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.600615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.600579 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.600615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.600596 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.605585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.605559 4990 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.605669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.605629 4990 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.700968 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.701000 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.701016 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.701062 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a30bdfa789f3bccffc4563d5bc301a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" (UID: \"c1a30bdfa789f3bccffc4563d5bc301a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.701077 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:52.701177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:52.701083 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5e8abc4535060f5a3a432c3ab1497e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" (UID: \"a5e8abc4535060f5a3a432c3ab1497e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:53.350370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.350336 4990 apiserver.go:52] "Watching apiserver" Oct 30 19:04:53.414417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.414389 4990 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:04:53.414819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.414797 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-l9rzn","openshift-dns/dns-default-lxr6q","openshift-network-operator/iptables-alerter-lzg24","kube-system/konnectivity-agent-5cp96","openshift-insights/insights-runtime-extractor-dw5vv","openshift-multus/multus-p2579","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal","openshift-monitoring/node-exporter-z5m5f","openshift-cluster-node-tuning-operator/tuned-fkzfj","kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj","openshift-ingress-canary/ingress-canary-fxcl2","kube-system/global-pull-secret-syncer-9kwwp","openshift-multus/multus-additional-cni-plugins-6txgd","openshift-network-diagnostics/network-check-target-nhhzq","openshift-image-registry/node-ca-zhtch","openshift-ovn-kubernetes/ovnkube-node-8mt7n","openshift-dns/node-resolver-s4kdn"] Oct 30 19:04:53.428500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428465 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428588 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:04:53.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428590 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:04:53.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428610 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428610 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428590 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.428951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.428936 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:04:53.429231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429217 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\"" Oct 30 19:04:53.429403 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429388 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\"" Oct 30 19:04:53.429476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429448 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\"" Oct 30 19:04:53.429476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429465 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\"" Oct 30 19:04:53.429571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429535 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:04:53.429619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429579 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\"" Oct 30 19:04:53.429678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.429649 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:04:53.437590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.437572 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:04:53.437590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.437587 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\"" Oct 30 19:04:53.437724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.437621 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.441834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.441798 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\"" Oct 30 19:04:53.441945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.441928 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\"" Oct 30 19:04:53.442540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.442518 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\"" Oct 30 19:04:53.442832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.442799 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:04:53.442904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.442874 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:04:53.443035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443006 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\"" Oct 30 19:04:53.443155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443056 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbgr5\"" Oct 30 19:04:53.443210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443161 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:04:53.443548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443530 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:04:53.443644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443549 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9w5x\"" Oct 30 19:04:53.443644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443558 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:04:53.443644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443621 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\"" Oct 30 19:04:53.443866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443848 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\"" Oct 30 19:04:53.443941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443851 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sz8sw\"" Oct 30 19:04:53.443941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443889 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bwrqq\"" Oct 30 19:04:53.443941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.443909 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:04:53.444076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.444052 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:04:53.445196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.444538 4990 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:53.445196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.445135 4990 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:53.445622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.445604 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:04:53.445999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.445981 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.445999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.445963 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:04:53.446168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446036 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.446168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446062 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:04:53.446266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446169 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.446266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446177 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:04:53.446266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446208 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.446266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446170 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:04:53.446558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446255 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.446605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446569 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.446605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446569 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.446674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446624 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.446715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.446686 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.449582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.449565 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.449803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.449790 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.451193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.451177 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.452038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452021 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.452118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452051 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.452172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452115 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:04:53.452172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452119 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:04:53.452172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452131 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.452172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452120 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:04:53.452172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452165 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:04:53.452340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452177 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:04:53.452340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452166 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:04:53.452340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452198 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:04:53.452416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.452365 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:04:53.461964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.461943 4990 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:04:53.465961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.465945 4990 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:53.466041 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:53.465984 4990 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-68.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" Oct 30 19:04:53.466657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.466641 4990 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:53.466737 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:04:53.466683 4990 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-68.ec2.internal" Oct 30 19:04:53.505394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505369 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.505496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505398 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.505496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505422 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.505496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505448 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.505600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505496 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2hs\" (UniqueName: \"kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.505660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505641 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.505722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505708 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.505771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505740 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.505839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505822 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.505900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505859 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.505900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505889 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.506000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505917 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.506000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505943 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.506000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505967 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.506000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.505991 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506077 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506109 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506073 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506140 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506171 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506222 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9712bd0c-c603-4d44-a99b-32bd83dc414c-config-volume\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506268 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506313 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506365 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506389 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-script-lib\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506398 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506398 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506429 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.506455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506457 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506465 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-etc-tuned\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506488 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506494 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovnkube-config\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506526 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506564 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506599 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplfk\" (UniqueName: \"kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506642 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506668 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506686 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506702 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc45\" (UniqueName: \"kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506717 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506756 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506789 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrgr\" (UniqueName: \"kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506814 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rtt\" (UniqueName: \"kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506839 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.506856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506862 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506910 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506940 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506954 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506963 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506980 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.506995 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507068 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ef107f8-59da-48f3-81dc-df4851b26044-tmp\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507119 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v998c\" (UniqueName: \"kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507140 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507154 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507170 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chshm\" (UniqueName: \"kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507170 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bc3361-f8d3-49de-8490-baf98084ac36-metrics-client-ca\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507186 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-original-pull-secret\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507204 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507250 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507290 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507306 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9712bd0c-c603-4d44-a99b-32bd83dc414c-metrics-tls\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507351 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507377 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507408 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cni-binary-copy\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507429 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507454 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507455 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507473 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507490 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507515 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.507584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507531 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507617 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507697 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507742 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd2c4560-125c-4c15-a15e-9e93a59e9a06-agent-certs\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507776 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507815 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-tmp-dir\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507818 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-metrics-certs\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507854 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507892 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.507986 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98tv\" (UniqueName: \"kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508052 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9712bd0c-c603-4d44-a99b-32bd83dc414c-tmp-dir\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508080 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508126 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508166 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508184 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508198 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508216 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508237 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508294 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508355 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5nk\" (UniqueName: \"kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508397 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508413 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508441 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-multus-daemon-config\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508451 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508476 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61da23fa-4dc5-4973-af62-c338ecdff26d-iptables-alerter-script\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.508473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508481 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508573 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508604 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-data-volume\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508613 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508631 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-tls\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508676 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508730 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508731 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd2c4560-125c-4c15-a15e-9e93a59e9a06-konnectivity-ca\") pod \"konnectivity-agent-5cp96\" (UID: \"cd2c4560-125c-4c15-a15e-9e93a59e9a06\") " pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508756 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67705ac6-bf6a-4e9e-9749-691459286cde-cni-binary-copy\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508771 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508801 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508833 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508881 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508904 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-env-overrides\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508929 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnc4\" (UniqueName: \"kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508932 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-serviceca\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.508961 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-ovn-node-metrics-cert\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509013 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509059 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509102 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509103 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509138 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509160 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509176 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509230 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509260 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq695\" (UniqueName: \"kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509281 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509321 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e563a40-316b-454d-8e73-fa7d8fd091b3-cert\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509344 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509354 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509394 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-textfile\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.509516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.509429 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.520376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.520359 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2hs\" (UniqueName: \"kubernetes.io/projected/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-kube-api-access-fz2hs\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.520749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.520732 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxq5\" (UniqueName: \"kubernetes.io/projected/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kube-api-access-frxq5\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.521126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.521106 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrgs6\" (UniqueName: \"kubernetes.io/projected/e8b22f35-75f3-4149-a2aa-5ee5074fa474-kube-api-access-zrgs6\") pod \"network-check-target-nhhzq\" (UID: \"e8b22f35-75f3-4149-a2aa-5ee5074fa474\") " pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:04:53.521838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.521824 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrgr\" (UniqueName: \"kubernetes.io/projected/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-kube-api-access-jsrgr\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.522446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.522426 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplfk\" (UniqueName: \"kubernetes.io/projected/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-kube-api-access-zplfk\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.522566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.522550 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chshm\" (UniqueName: \"kubernetes.io/projected/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-kube-api-access-chshm\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.522634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.522617 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98tv\" (UniqueName: \"kubernetes.io/projected/9712bd0c-c603-4d44-a99b-32bd83dc414c-kube-api-access-t98tv\") pod \"dns-default-lxr6q\" (UID: \"9712bd0c-c603-4d44-a99b-32bd83dc414c\") " pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:53.522681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.522637 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rtt\" (UniqueName: \"kubernetes.io/projected/4e563a40-316b-454d-8e73-fa7d8fd091b3-kube-api-access-x5rtt\") pod \"ingress-canary-fxcl2\" (UID: \"4e563a40-316b-454d-8e73-fa7d8fd091b3\") " pod="openshift-ingress-canary/ingress-canary-fxcl2" Oct 30 19:04:53.522948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.522930 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsxw\" (UniqueName: \"kubernetes.io/projected/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-kube-api-access-wwsxw\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.523126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.523109 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v998c\" (UniqueName: \"kubernetes.io/projected/6ef107f8-59da-48f3-81dc-df4851b26044-kube-api-access-v998c\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.523226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.523211 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnc4\" (UniqueName: \"kubernetes.io/projected/61da23fa-4dc5-4973-af62-c338ecdff26d-kube-api-access-6rnc4\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.523467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.523451 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5nk\" (UniqueName: \"kubernetes.io/projected/67705ac6-bf6a-4e9e-9749-691459286cde-kube-api-access-bn5nk\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.523629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.523616 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc45\" (UniqueName: \"kubernetes.io/projected/53bc3361-f8d3-49de-8490-baf98084ac36-kube-api-access-9jc45\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.524056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.524042 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq695\" (UniqueName: \"kubernetes.io/projected/ed069dac-23ef-4ec7-88a9-e74dc4e4161b-kube-api-access-vq695\") pod \"network-metrics-daemon-l9rzn\" (UID: \"ed069dac-23ef-4ec7-88a9-e74dc4e4161b\") " pod="openshift-multus/network-metrics-daemon-l9rzn" Oct 30 19:04:53.609889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609823 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.609889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609865 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.609889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609881 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609904 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609927 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609943 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-conf-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609963 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-host\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609964 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-etc-kubernetes\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609993 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-socket-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.609967 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-slash\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610040 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610059 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610074 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610105 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610131 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610134 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-netd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610141 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610155 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610180 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610181 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-systemd\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610194 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-multus-socket-dir-parent\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610210 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610225 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab-crio-socket\") pod \"insights-runtime-extractor-dw5vv\" (UID: \"ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab\") " pod="openshift-insights/insights-runtime-extractor-dw5vv" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610239 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5b7c1f1-3b2d-4b73-8870-f1cffe60206e-hosts-file\") pod \"node-resolver-s4kdn\" (UID: \"e5b7c1f1-3b2d-4b73-8870-f1cffe60206e\") " pod="openshift-dns/node-resolver-s4kdn" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610229 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610273 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-sys\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610248 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-systemd-units\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610292 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d9d8a4b-34d0-47ba-abc1-8e71a55d600c-host\") pod \"node-ca-zhtch\" (UID: \"8d9d8a4b-34d0-47ba-abc1-8e71a55d600c\") " pod="openshift-image-registry/node-ca-zhtch" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610285 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610324 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610343 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610364 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610376 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610387 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-lib-modules\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610379 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610407 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-netns\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.610388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610427 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610454 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610456 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61da23fa-4dc5-4973-af62-c338ecdff26d-host-slash\") pod \"iptables-alerter-lzg24\" (UID: \"61da23fa-4dc5-4973-af62-c338ecdff26d\") " pod="openshift-network-operator/iptables-alerter-lzg24" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610476 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610503 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-root\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610524 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-kubernetes\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610525 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610568 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-sys\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610573 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610600 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610624 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610569 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-system-cni-dir\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610646 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610672 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610675 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-var-lib-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610681 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610673 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-node-log\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610697 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610704 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-bin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610709 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-run\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610721 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-kubelet\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610723 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610749 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610770 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610773 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-systemd\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610792 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610818 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-system-cni-dir\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610819 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610833 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-var-lib-kubelet\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610848 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610854 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-multus-certs\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610869 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-cni-bin\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610910 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610926 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-ovn\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610956 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610970 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.610997 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611022 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-device-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611071 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-log-socket\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611075 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611116 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-hostroot\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611131 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611150 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611148 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-registration-dir\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611164 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611179 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611193 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-netns\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611178 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611202 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611235 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611260 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611204 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-etc-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611201 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611299 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e6381-5d5c-4d33-86cb-3e528c197bcd-cnibin\") pod \"multus-additional-cni-plugins-6txgd\" (UID: \"5d5e6381-5d5c-4d33-86cb-3e528c197bcd\") " pod="openshift-multus/multus-additional-cni-plugins-6txgd" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611275 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-modprobe-d\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611311 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-cnibin\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611317 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysctl-conf\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611327 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611364 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611388 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611393 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-run-k8s-cni-cncf-io\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611415 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611439 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611452 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-cni-multus\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611467 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611466 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611491 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-kubelet-config\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611491 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ef107f8-59da-48f3-81dc-df4851b26044-etc-sysconfig\") pod \"tuned-fkzfj\" (UID: \"6ef107f8-59da-48f3-81dc-df4851b26044\") " pod="openshift-cluster-node-tuning-operator/tuned-fkzfj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611497 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611528 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc2715ff-fc4b-46c6-89fe-f95275cfe46d-dbus\") pod \"global-pull-secret-syncer-9kwwp\" (UID: \"fc2715ff-fc4b-46c6-89fe-f95275cfe46d\") " pod="kube-system/global-pull-secret-syncer-9kwwp" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611583 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611548 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53bc3361-f8d3-49de-8490-baf98084ac36-node-exporter-wtmp\") pod \"node-exporter-z5m5f\" (UID: \"53bc3361-f8d3-49de-8490-baf98084ac36\") " pod="openshift-monitoring/node-exporter-z5m5f" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611507 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fd9e114-e938-4928-94e8-c9c3bd7ae37b-run-openvswitch\") pod \"ovnkube-node-8mt7n\" (UID: \"4fd9e114-e938-4928-94e8-c9c3bd7ae37b\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611606 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67705ac6-bf6a-4e9e-9749-691459286cde-host-var-lib-kubelet\") pod \"multus-p2579\" (UID: \"67705ac6-bf6a-4e9e-9749-691459286cde\") " pod="openshift-multus/multus-p2579" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611622 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:53.611679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:53.611671 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e066ed65-096a-4d83-bcf0-cc3cb10267fc-sys-fs\") pod \"aws-ebs-csi-driver-node-s6chj\" (UID: \"e066ed65-096a-4d83-bcf0-cc3cb10267fc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6chj" Oct 30 19:04:54.629678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:54.629647 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lxr6q" Oct 30 19:04:55.423464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:55.423431 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:04:55.786688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:55.786596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:55.952544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:04:55.952510 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:00.707976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:00.707946 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:05:00.708462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:00.708192 4990 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:05:01.291248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:01.291216 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:05:05.093681 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:05:05.093598 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:05:07.037214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:07.037170 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:07.171457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:07.171418 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:07.484546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:07.484516 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5cp96" Oct 30 19:05:08.111656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:08.111620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:14.441225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:14.441199 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mt7n" Oct 30 19:05:18.991703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:18.991658 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:19.152542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:19.152502 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:21.294017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:21.293988 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nhhzq" Oct 30 19:05:21.603929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:21.603888 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:21.892861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:21.892771 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:24.636877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.636849 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:05:24.679862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.679825 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:05:24.680012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.679949 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.692676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.692651 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:05:24.692676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.692655 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:05:24.692860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.692676 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:05:24.692860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.692811 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:05:24.702619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.702599 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:05:24.702746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.702624 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:05:24.702746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.702671 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:05:24.702858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.702753 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z7mb7\"" Oct 30 19:05:24.703010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.702998 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:05:24.713028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.713011 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:05:24.799842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799808 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.799972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799849 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.799972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799872 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.799972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799960 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-config-volume\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799976 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-web-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.799999 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800019 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kk78\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-kube-api-access-2kk78\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800036 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-config-out\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800084 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800169 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800201 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800251 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.800447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.800323 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901399 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901441 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901489 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901515 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901540 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901566 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901594 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901621 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-config-volume\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901650 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-web-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901687 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901715 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kk78\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-kube-api-access-2kk78\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901743 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-config-out\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901774 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901849 4990 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:05:24.901993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.901983 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.902514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.902486 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.902922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.902900 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f374c3-876b-413d-8b54-daea5cc60676-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.904817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.904793 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.904817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.904815 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.904974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.904875 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.905235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.905216 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.905301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.905244 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.905406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.905378 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.905564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.905543 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-config-volume\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.905622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.905608 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47f374c3-876b-413d-8b54-daea5cc60676-web-config\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.906672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.906656 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47f374c3-876b-413d-8b54-daea5cc60676-config-out\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:24.921907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:24.921888 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kk78\" (UniqueName: \"kubernetes.io/projected/47f374c3-876b-413d-8b54-daea5cc60676-kube-api-access-2kk78\") pod \"alertmanager-main-0\" (UID: \"47f374c3-876b-413d-8b54-daea5cc60676\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:25.087983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:25.087950 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:05:25.411647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:25.411617 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:05:25.416868 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:05:25.416838 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f374c3_876b_413d_8b54_daea5cc60676.slice/crio-b055b1bb2c1c95ac4f4102b7180db9225ac626d423dca90e02b0b04e21e648bc WatchSource:0}: Error finding container b055b1bb2c1c95ac4f4102b7180db9225ac626d423dca90e02b0b04e21e648bc: Status 404 returned error can't find the container with id b055b1bb2c1c95ac4f4102b7180db9225ac626d423dca90e02b0b04e21e648bc Oct 30 19:05:25.418805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:25.418789 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:05:25.536608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:25.536575 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"b055b1bb2c1c95ac4f4102b7180db9225ac626d423dca90e02b0b04e21e648bc"} Oct 30 19:05:26.540914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.540883 4990 generic.go:356] "Generic (PLEG): container finished" podID="47f374c3-876b-413d-8b54-daea5cc60676" containerID="c8fc029eb25c13786450d4837b7215254e25133df74454c37e1c3f965b13d069" exitCode=0 Oct 30 19:05:26.541380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.540932 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerDied","Data":"c8fc029eb25c13786450d4837b7215254e25133df74454c37e1c3f965b13d069"} Oct 30 19:05:26.836734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.836702 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-658d9f684f-ncjrn"] Oct 30 19:05:26.839236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.839213 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:26.852249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.852227 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 19:05:26.852249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.852236 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 19:05:26.852546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.852527 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 19:05:26.852626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.852563 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2l9xs\"" Oct 30 19:05:26.852626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.852570 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 19:05:26.866145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.866125 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-658d9f684f-ncjrn"] Oct 30 19:05:26.941731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.941708 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 19:05:26.952441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:26.952420 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 19:05:27.019703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.019675 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.019821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.019738 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-federate-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.019821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.019805 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-serving-certs-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.019915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.019892 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.020030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.020003 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-metrics-client-ca\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.020172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.020046 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.020172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.020085 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsp8\" (UniqueName: \"kubernetes.io/projected/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-kube-api-access-lmsp8\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.020252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.020199 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121034 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-serving-certs-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121074 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121120 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-metrics-client-ca\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121140 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121164 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsp8\" (UniqueName: \"kubernetes.io/projected/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-kube-api-access-lmsp8\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121207 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121242 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.121393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121287 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-federate-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.122123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121974 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-metrics-client-ca\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.122123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.121990 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-serving-certs-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.122299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.122217 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.124186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.124158 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.124500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.124478 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-secret-telemeter-client\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.124584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.124513 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-telemeter-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.124584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.124553 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-federate-client-tls\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.137965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.137944 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsp8\" (UniqueName: \"kubernetes.io/projected/6a6b7bd2-39e2-484b-9e37-ac5bad17363a-kube-api-access-lmsp8\") pod \"telemeter-client-658d9f684f-ncjrn\" (UID: \"6a6b7bd2-39e2-484b-9e37-ac5bad17363a\") " pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.150835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.150816 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" Oct 30 19:05:27.479350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.479274 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-658d9f684f-ncjrn"] Oct 30 19:05:27.484372 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:05:27.484273 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6b7bd2_39e2_484b_9e37_ac5bad17363a.slice/crio-c4b69b0b2878bbd2705ed86c9ec8d4b2f243fc64816845e144b3b9d73887d194 WatchSource:0}: Error finding container c4b69b0b2878bbd2705ed86c9ec8d4b2f243fc64816845e144b3b9d73887d194: Status 404 returned error can't find the container with id c4b69b0b2878bbd2705ed86c9ec8d4b2f243fc64816845e144b3b9d73887d194 Oct 30 19:05:27.545384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:27.545352 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" event={"ID":"6a6b7bd2-39e2-484b-9e37-ac5bad17363a","Type":"ContainerStarted","Data":"c4b69b0b2878bbd2705ed86c9ec8d4b2f243fc64816845e144b3b9d73887d194"} Oct 30 19:05:28.551689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:28.551662 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"4e095a60d181bc838856bd27baeab0e56c947ab80115f4f9f04fdadba3ecc697"} Oct 30 19:05:28.552048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:28.551729 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"7a8bcc1fbaa334ef13e9008a248dd59bcbc00db2146a8714005aca80159578b4"} Oct 30 19:05:28.552048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:28.551753 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"f8d5f0bc148b70da7a1f45c8d9c825d6c701367fa10b2523116c7bf511eaa6f9"} Oct 30 19:05:28.552048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:28.551778 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"ed60c16def052afb8ac13d7c857380f5bf39abb9b54ade7dac8779e1c0702b0a"} Oct 30 19:05:28.552048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:28.551799 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"43a7beb14769bf3882ab192a65def380424dfeb1088db3394e339b4c28c9fd98"} Oct 30 19:05:29.558214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.558180 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47f374c3-876b-413d-8b54-daea5cc60676","Type":"ContainerStarted","Data":"9d2a0d3c65cfe6165824bd765db720671d885ec5773393c3b5d7fa04097baf99"} Oct 30 19:05:29.560102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.560069 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" event={"ID":"6a6b7bd2-39e2-484b-9e37-ac5bad17363a","Type":"ContainerStarted","Data":"8a088d75a637f8daee35619da75e14435f91aae7d87f4b916677128b5fdf313a"} Oct 30 19:05:29.560208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.560121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" event={"ID":"6a6b7bd2-39e2-484b-9e37-ac5bad17363a","Type":"ContainerStarted","Data":"eddcc4d6fd654d075c5fa6eafc9989c83c63b8e68d29565827923d15d3389d46"} Oct 30 19:05:29.560208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.560140 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" event={"ID":"6a6b7bd2-39e2-484b-9e37-ac5bad17363a","Type":"ContainerStarted","Data":"cdbf44f0b87bfdc76488a8855f418d732403855a9f6267eb4656d5ad8d483ca8"} Oct 30 19:05:29.631622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.631583 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4963851679999998 podStartE2EDuration="5.631572364s" totalImagesPullingTime="3.122436965s" totalInitContainerRuntime="12.750231ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:29.631321797 +0000 UTC m=+37.342847163" watchObservedRunningTime="2026-10-30 19:05:29.631572364 +0000 UTC m=+37.343097721" Oct 30 19:05:29.672212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:29.672167 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-658d9f684f-ncjrn" podStartSLOduration=2.058003925 podStartE2EDuration="3.67215397s" totalImagesPullingTime="1.614150045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:29.671057047 +0000 UTC m=+37.382582413" watchObservedRunningTime="2026-10-30 19:05:29.67215397 +0000 UTC m=+37.383679338" Oct 30 19:05:30.030960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.030871 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.176175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.176130 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.328006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.327976 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.466799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.466758 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.694078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.693983 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.877116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:30.877048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:35.093541 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:05:35.093497 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:05:55.908575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:05:55.908541 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:05.094870 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:06:05.094831 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:06:05.424417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:05.424328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:35.096859 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:06:35.096767 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:06:36.759480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:36.759440 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:37.474905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:37.474870 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:37.615664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:37.615627 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:41.806011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:41.805973 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:41.942415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:41.942379 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:42.115919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:42.115836 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:42.255223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:42.255194 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.435920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:50.435887 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.567995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:50.567957 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:56.023838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:06:56.023803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:05.096019 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:07:05.095975 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:07:07.045616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:07:07.045583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:07.189492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:07:07.189457 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:35.094897 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:07:35.094860 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:07:56.157148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:07:56.157112 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.095215 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:08:05.095133 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:08:35.093832 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:08:35.093794 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:08:56.276576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:08:56.276543 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.094707 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:09:05.094669 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:09:19.596646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:19.596610 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.342271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:20.342236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:32.768684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:32.768636 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:33.856612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:33.856576 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:35.096154 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:09:35.096116 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:09:49.419701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:49.419670 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:49.612329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:49.612295 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:50.282680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.282649 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l"] Oct 30 19:09:50.285011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.284990 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.297706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.297680 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8ss\" (UniqueName: \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.297812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.297715 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.297812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.297796 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.306990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.306967 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:09:50.306990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.306983 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kp9j6\"" Oct 30 19:09:50.307124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.307056 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:09:50.312801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.312781 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l"] Oct 30 19:09:50.398738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.398705 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.398907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.398747 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8ss\" (UniqueName: \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.398907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.398769 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.399183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.399162 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.399225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.399165 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.416125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.416083 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8ss\" (UniqueName: \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.597024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.596992 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:09:50.887432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:50.887348 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l"] Oct 30 19:09:50.892715 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:09:50.892688 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f6e335_5ec7_4638_a9d3_9ad719491a9e.slice/crio-82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260 WatchSource:0}: Error finding container 82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260: Status 404 returned error can't find the container with id 82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260 Oct 30 19:09:51.361458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:51.361424 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerStarted","Data":"82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260"} Oct 30 19:09:52.409279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:52.409245 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:09:52.413532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:52.413498 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:09:56.392028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:56.391986 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:57.383758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:57.383725 4990 generic.go:356] "Generic (PLEG): container finished" podID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerID="912cd2e4f5b9d4e30ed80c0eaf31bf646189376a56b4391b4d3c25339fd14c3a" exitCode=0 Oct 30 19:09:57.383945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:09:57.383800 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerDied","Data":"912cd2e4f5b9d4e30ed80c0eaf31bf646189376a56b4391b4d3c25339fd14c3a"} Oct 30 19:10:00.395485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:00.395448 4990 generic.go:356] "Generic (PLEG): container finished" podID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerID="73365321d07a90504dfa32a812ebf3c449e2199c0abd9f8fa2ef8f3d6f35dad0" exitCode=0 Oct 30 19:10:00.396643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:00.396521 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerDied","Data":"73365321d07a90504dfa32a812ebf3c449e2199c0abd9f8fa2ef8f3d6f35dad0"} Oct 30 19:10:05.098639 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:10:05.098600 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:10:06.420998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:06.420964 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerStarted","Data":"eb513312253a0d2aae974a97ecd4abcd4f6198552941e4453fbf964f6cc47c14"} Oct 30 19:10:06.455183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:06.455133 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" podStartSLOduration=3.24396001 podStartE2EDuration="16.455121971s" totalImagesPullingTime="13.211161961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:06.454794339 +0000 UTC m=+314.166319710" watchObservedRunningTime="2026-10-30 19:10:06.455121971 +0000 UTC m=+314.166647334" Oct 30 19:10:10.018309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:10.018276 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:12.940934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:12.940897 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:13.084374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:13.084335 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:16.458151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:16.458056 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:10:16.458831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:16.458809 4990 generic.go:356] "Generic (PLEG): container finished" podID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerID="eb513312253a0d2aae974a97ecd4abcd4f6198552941e4453fbf964f6cc47c14" exitCode=1 Oct 30 19:10:16.458902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:16.458862 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerDied","Data":"eb513312253a0d2aae974a97ecd4abcd4f6198552941e4453fbf964f6cc47c14"} Oct 30 19:10:17.581263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.581233 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:10:17.596011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.595978 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:10:17.690465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.690429 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\" (UniqueName: \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\") pod \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " Oct 30 19:10:17.690646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.690509 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\") pod \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " Oct 30 19:10:17.690646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.690609 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\") pod \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\" (UID: \"01f6e335-5ec7-4638-a9d3-9ad719491a9e\") " Oct 30 19:10:17.691303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.691275 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle" pod "01f6e335-5ec7-4638-a9d3-9ad719491a9e" (UID: "01f6e335-5ec7-4638-a9d3-9ad719491a9e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:17.692660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.692637 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss" pod "01f6e335-5ec7-4638-a9d3-9ad719491a9e" (UID: "01f6e335-5ec7-4638-a9d3-9ad719491a9e"). InnerVolumeSpecName "kube-api-access-xc8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:17.698017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.697984 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util" pod "01f6e335-5ec7-4638-a9d3-9ad719491a9e" (UID: "01f6e335-5ec7-4638-a9d3-9ad719491a9e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:17.791838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.791750 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:17.791838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.791782 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f6e335-5ec7-4638-a9d3-9ad719491a9e-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:17.791838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:17.791794 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8ss\" (UniqueName: \"kubernetes.io/projected/01f6e335-5ec7-4638-a9d3-9ad719491a9e-kube-api-access-xc8ss\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:18.466546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:18.466516 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:10:18.467296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:18.467280 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" Oct 30 19:10:18.467380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:18.467279 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l" event={"ID":"01f6e335-5ec7-4638-a9d3-9ad719491a9e","Type":"ContainerDied","Data":"82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260"} Oct 30 19:10:18.467482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:18.467396 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82f1aaa990aeafe3817f2c232a44a02fed64a0e4fdc4c5bedd60ac542b748260" Oct 30 19:10:26.025068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025030 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr"] Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025387 4990 cpu_manager.go:414] "containerMap: removing container" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="extract" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025406 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="extract" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025425 4990 cpu_manager.go:414] "containerMap: removing container" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="pull" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025433 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="pull" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025456 4990 cpu_manager.go:414] "containerMap: removing container" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="util" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025463 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="util" Oct 30 19:10:26.025657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.025533 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f6e335-5ec7-4638-a9d3-9ad719491a9e" containerName="extract" Oct 30 19:10:26.028107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.028067 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.050560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.050528 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:10:26.050728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.050608 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:10:26.050728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.050714 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kp9j6\"" Oct 30 19:10:26.051007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.050982 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.051106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.051044 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.051106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.051080 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzzv\" (UniqueName: \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.051592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.051568 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr"] Oct 30 19:10:26.151661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.151616 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.151842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.151669 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.151842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.151694 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzzv\" (UniqueName: \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.152047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.152026 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.152085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.152065 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.168589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.168561 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzzv\" (UniqueName: \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.340631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.340598 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:26.631068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:26.630990 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr"] Oct 30 19:10:26.637194 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:26.637162 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e65710_556b_4bb7_ad71_89b3b1c51cda.slice/crio-cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744 WatchSource:0}: Error finding container cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744: Status 404 returned error can't find the container with id cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744 Oct 30 19:10:27.495567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:27.495478 4990 generic.go:356] "Generic (PLEG): container finished" podID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerID="99722d58aa741266e462b98b39f8ea14426b41c1f81b068fb9c93a90e4a421a0" exitCode=0 Oct 30 19:10:27.495567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:27.495534 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" event={"ID":"e0e65710-556b-4bb7-ad71-89b3b1c51cda","Type":"ContainerDied","Data":"99722d58aa741266e462b98b39f8ea14426b41c1f81b068fb9c93a90e4a421a0"} Oct 30 19:10:27.495971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:27.495578 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" event={"ID":"e0e65710-556b-4bb7-ad71-89b3b1c51cda","Type":"ContainerStarted","Data":"cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744"} Oct 30 19:10:28.500725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:28.500691 4990 generic.go:356] "Generic (PLEG): container finished" podID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerID="5f38666c7d14e3cdc78bb7354fe3ca528f91f9cb9d9d8535d51202ff11912919" exitCode=0 Oct 30 19:10:28.501141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:28.500747 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" event={"ID":"e0e65710-556b-4bb7-ad71-89b3b1c51cda","Type":"ContainerDied","Data":"5f38666c7d14e3cdc78bb7354fe3ca528f91f9cb9d9d8535d51202ff11912919"} Oct 30 19:10:29.505788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:29.505751 4990 generic.go:356] "Generic (PLEG): container finished" podID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerID="a558eceecb65717c1fe862dea402f371161e8025dde2216328c5f855d47dc1a4" exitCode=0 Oct 30 19:10:29.506202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:29.505829 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" event={"ID":"e0e65710-556b-4bb7-ad71-89b3b1c51cda","Type":"ContainerDied","Data":"a558eceecb65717c1fe862dea402f371161e8025dde2216328c5f855d47dc1a4"} Oct 30 19:10:30.632826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.632800 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:30.685043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.685011 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\") pod \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " Oct 30 19:10:30.685256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.685052 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\") pod \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " Oct 30 19:10:30.685256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.685109 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\" (UniqueName: \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\") pod \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\" (UID: \"e0e65710-556b-4bb7-ad71-89b3b1c51cda\") " Oct 30 19:10:30.685752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.685720 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle" pod "e0e65710-556b-4bb7-ad71-89b3b1c51cda" (UID: "e0e65710-556b-4bb7-ad71-89b3b1c51cda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:30.687263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.687231 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv" pod "e0e65710-556b-4bb7-ad71-89b3b1c51cda" (UID: "e0e65710-556b-4bb7-ad71-89b3b1c51cda"). InnerVolumeSpecName "kube-api-access-7vzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:30.690971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.690931 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util" pod "e0e65710-556b-4bb7-ad71-89b3b1c51cda" (UID: "e0e65710-556b-4bb7-ad71-89b3b1c51cda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:30.786587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.786500 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:30.786587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.786532 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0e65710-556b-4bb7-ad71-89b3b1c51cda-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:30.786587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:30.786541 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzzv\" (UniqueName: \"kubernetes.io/projected/e0e65710-556b-4bb7-ad71-89b3b1c51cda-kube-api-access-7vzzv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:31.513402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:31.513363 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" event={"ID":"e0e65710-556b-4bb7-ad71-89b3b1c51cda","Type":"ContainerDied","Data":"cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744"} Oct 30 19:10:31.513575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:31.513410 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4p6sr" Oct 30 19:10:31.513575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:31.513417 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd9a6c71e58949a46f66bd058b04aa34f6119bcd7c8699fea2326e2ef98f5744" Oct 30 19:10:35.094992 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:10:35.094957 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:10:36.521196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:36.521163 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:37.377630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:37.377593 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:38.513175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513136 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm"] Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513382 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="pull" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513393 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="pull" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513402 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="extract" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513407 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="extract" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513422 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="util" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513427 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="util" Oct 30 19:10:38.513565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.513471 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e65710-556b-4bb7-ad71-89b3b1c51cda" containerName="extract" Oct 30 19:10:38.515223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.515206 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.540060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.540031 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 19:10:38.540060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.540031 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:10:38.540264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.540068 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-j6nsb\"" Oct 30 19:10:38.540833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.540813 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm"] Oct 30 19:10:38.640767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.640729 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz74k\" (UniqueName: \"kubernetes.io/projected/eb204899-f709-49d3-bc3a-d56b7b848c87-kube-api-access-vz74k\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.640957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.640773 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb204899-f709-49d3-bc3a-d56b7b848c87-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.742226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.742190 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz74k\" (UniqueName: \"kubernetes.io/projected/eb204899-f709-49d3-bc3a-d56b7b848c87-kube-api-access-vz74k\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.742226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.742230 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb204899-f709-49d3-bc3a-d56b7b848c87-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.742667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.742648 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb204899-f709-49d3-bc3a-d56b7b848c87-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.759787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.759759 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz74k\" (UniqueName: \"kubernetes.io/projected/eb204899-f709-49d3-bc3a-d56b7b848c87-kube-api-access-vz74k\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-4m8lm\" (UID: \"eb204899-f709-49d3-bc3a-d56b7b848c87\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:38.825699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:38.825654 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" Oct 30 19:10:39.145555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:39.145456 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm"] Oct 30 19:10:39.150497 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:39.150466 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb204899_f709_49d3_bc3a_d56b7b848c87.slice/crio-a04c985bfb1f9cad9600fdf84d8b180ff85331e8d4d477f8aeb3260fec668ffb WatchSource:0}: Error finding container a04c985bfb1f9cad9600fdf84d8b180ff85331e8d4d477f8aeb3260fec668ffb: Status 404 returned error can't find the container with id a04c985bfb1f9cad9600fdf84d8b180ff85331e8d4d477f8aeb3260fec668ffb Oct 30 19:10:39.153010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:39.152989 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:10:39.539516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:39.539428 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" event={"ID":"eb204899-f709-49d3-bc3a-d56b7b848c87","Type":"ContainerStarted","Data":"a04c985bfb1f9cad9600fdf84d8b180ff85331e8d4d477f8aeb3260fec668ffb"} Oct 30 19:10:39.731459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:39.731422 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:39.883962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:39.883919 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:41.549881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:41.549841 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" event={"ID":"eb204899-f709-49d3-bc3a-d56b7b848c87","Type":"ContainerStarted","Data":"bd40c4eb9c2e852c0336da704249c27779739fe57875617ecfdd22d087478311"} Oct 30 19:10:41.609919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:41.609868 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-4m8lm" podStartSLOduration=1.87384899 podStartE2EDuration="3.609853116s" totalImagesPullingTime="1.736004126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:41.599388079 +0000 UTC m=+349.310913445" watchObservedRunningTime="2026-10-30 19:10:41.609853116 +0000 UTC m=+349.321378479" Oct 30 19:10:41.719648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:41.719616 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:41.920052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:41.919962 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.188719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.188633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.363627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.363590 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.723320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.723288 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt"] Oct 30 19:10:42.725567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.725544 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.748619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.748591 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:10:42.748619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.748605 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:10:42.748780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.748634 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-gdxdw\"" Oct 30 19:10:42.749413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.749394 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt"] Oct 30 19:10:42.774473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.774444 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.774639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.774492 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4g8\" (UniqueName: \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.774639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.774608 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.831665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.831626 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cc594"] Oct 30 19:10:42.834251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.834229 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.846524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.846503 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:10:42.846645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.846591 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:10:42.846704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.846691 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jlnc6\"" Oct 30 19:10:42.855249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.855230 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cc594"] Oct 30 19:10:42.875403 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875367 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4g8\" (UniqueName: \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.875525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875413 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsqb\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-kube-api-access-2gsqb\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.875525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875476 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.875525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875507 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.875699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875677 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.875868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875850 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.875972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.875957 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.893459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.893435 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4g8\" (UniqueName: \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:42.976493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.976415 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.976493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.976473 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsqb\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-kube-api-access-2gsqb\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.993730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.993698 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsqb\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-kube-api-access-2gsqb\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:42.993855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:42.993764 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b26ecda-af1c-4159-a872-4bdec846ad44-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cc594\" (UID: \"7b26ecda-af1c-4159-a872-4bdec846ad44\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:43.042424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.042390 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:43.152590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.152553 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:43.334124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.334073 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt"] Oct 30 19:10:43.338989 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:43.338961 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6f6903_6471_4f86_b283_c154510d1f34.slice/crio-485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609 WatchSource:0}: Error finding container 485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609: Status 404 returned error can't find the container with id 485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609 Oct 30 19:10:43.441770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.441743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cc594"] Oct 30 19:10:43.485969 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:43.485937 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b26ecda_af1c_4159_a872_4bdec846ad44.slice/crio-7f051cfde90571baab00336591514096d73d552de73b2630de7ae0e0c4783159 WatchSource:0}: Error finding container 7f051cfde90571baab00336591514096d73d552de73b2630de7ae0e0c4783159: Status 404 returned error can't find the container with id 7f051cfde90571baab00336591514096d73d552de73b2630de7ae0e0c4783159 Oct 30 19:10:43.559195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.559144 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" event={"ID":"7b26ecda-af1c-4159-a872-4bdec846ad44","Type":"ContainerStarted","Data":"7f051cfde90571baab00336591514096d73d552de73b2630de7ae0e0c4783159"} Oct 30 19:10:43.560510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.560480 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f6903-6471-4f86-b283-c154510d1f34" containerID="a9c41b64c14626db6c2986022ae62e9c13aaefc13661a9917fca6c49330d2719" exitCode=0 Oct 30 19:10:43.560638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.560518 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" event={"ID":"ef6f6903-6471-4f86-b283-c154510d1f34","Type":"ContainerDied","Data":"a9c41b64c14626db6c2986022ae62e9c13aaefc13661a9917fca6c49330d2719"} Oct 30 19:10:43.560638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.560558 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" event={"ID":"ef6f6903-6471-4f86-b283-c154510d1f34","Type":"ContainerStarted","Data":"485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609"} Oct 30 19:10:43.922577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:43.922546 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.347395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.347333 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.462208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.462180 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2"] Oct 30 19:10:44.464732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.464709 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.500362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.500338 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c2jb5\"" Oct 30 19:10:44.513344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.513317 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2"] Oct 30 19:10:44.592287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.592238 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m964\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-kube-api-access-5m964\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.592452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.592291 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.693044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.692963 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m964\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-kube-api-access-5m964\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.693044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.693011 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.710861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.710834 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.710994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.710956 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m964\" (UniqueName: \"kubernetes.io/projected/f4180748-0482-4546-9bbc-781ad1932d3f-kube-api-access-5m964\") pod \"cert-manager-cainjector-5c57bbd4fc-wghz2\" (UID: \"f4180748-0482-4546-9bbc-781ad1932d3f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:44.776767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:44.776728 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" Oct 30 19:10:45.079663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:45.079628 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2"] Oct 30 19:10:45.084728 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:45.084686 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4180748_0482_4546_9bbc_781ad1932d3f.slice/crio-6330a0f3202e8b7a664a5faac921d7e3ab4c0b55b9e8c7cc5eecfb9fb9519ee6 WatchSource:0}: Error finding container 6330a0f3202e8b7a664a5faac921d7e3ab4c0b55b9e8c7cc5eecfb9fb9519ee6: Status 404 returned error can't find the container with id 6330a0f3202e8b7a664a5faac921d7e3ab4c0b55b9e8c7cc5eecfb9fb9519ee6 Oct 30 19:10:45.578851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:45.578787 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" event={"ID":"f4180748-0482-4546-9bbc-781ad1932d3f","Type":"ContainerStarted","Data":"6330a0f3202e8b7a664a5faac921d7e3ab4c0b55b9e8c7cc5eecfb9fb9519ee6"} Oct 30 19:10:46.585574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:46.585539 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f6903-6471-4f86-b283-c154510d1f34" containerID="f93373a34ca93ea0f59bf83989f2bc492cbae390eb992ed6d205171856f7ffa5" exitCode=0 Oct 30 19:10:46.585910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:46.585633 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" event={"ID":"ef6f6903-6471-4f86-b283-c154510d1f34","Type":"ContainerDied","Data":"f93373a34ca93ea0f59bf83989f2bc492cbae390eb992ed6d205171856f7ffa5"} Oct 30 19:10:46.718852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:46.718821 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:46.876949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:46.876922 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:47.590658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.590620 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" event={"ID":"7b26ecda-af1c-4159-a872-4bdec846ad44","Type":"ContainerStarted","Data":"839c79546d01e608085b3b9cc9f29ef65027fab56ede21170c4e8a85579a8326"} Oct 30 19:10:47.591152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.590684 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:47.592155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.592130 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" event={"ID":"f4180748-0482-4546-9bbc-781ad1932d3f","Type":"ContainerStarted","Data":"b02a09f089c91ac9a8a24704ee8b42ae493b335dde03d30558eac1dd84ff1ed9"} Oct 30 19:10:47.594178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.594153 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f6903-6471-4f86-b283-c154510d1f34" containerID="1ff8aac1a2d226b991f706c67f6e2c59558cd58a8a811704dc73902c97a703e2" exitCode=0 Oct 30 19:10:47.594291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.594198 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" event={"ID":"ef6f6903-6471-4f86-b283-c154510d1f34","Type":"ContainerDied","Data":"1ff8aac1a2d226b991f706c67f6e2c59558cd58a8a811704dc73902c97a703e2"} Oct 30 19:10:47.626388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.626341 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" podStartSLOduration=2.545859297 podStartE2EDuration="5.626328706s" totalImagesPullingTime="3.080469409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:47.625914281 +0000 UTC m=+355.337439647" watchObservedRunningTime="2026-10-30 19:10:47.626328706 +0000 UTC m=+355.337854072" Oct 30 19:10:47.703536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:47.703487 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wghz2" podStartSLOduration=2.045457732 podStartE2EDuration="3.703470916s" totalImagesPullingTime="1.658013184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:47.703407063 +0000 UTC m=+355.414932429" watchObservedRunningTime="2026-10-30 19:10:47.703470916 +0000 UTC m=+355.414996285" Oct 30 19:10:48.717394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.717373 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:48.828626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.828598 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\") pod \"ef6f6903-6471-4f86-b283-c154510d1f34\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " Oct 30 19:10:48.828770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.828677 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\" (UniqueName: \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\") pod \"ef6f6903-6471-4f86-b283-c154510d1f34\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " Oct 30 19:10:48.828770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.828707 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\") pod \"ef6f6903-6471-4f86-b283-c154510d1f34\" (UID: \"ef6f6903-6471-4f86-b283-c154510d1f34\") " Oct 30 19:10:48.829139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.829112 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle" pod "ef6f6903-6471-4f86-b283-c154510d1f34" (UID: "ef6f6903-6471-4f86-b283-c154510d1f34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:48.830726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.830700 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8" pod "ef6f6903-6471-4f86-b283-c154510d1f34" (UID: "ef6f6903-6471-4f86-b283-c154510d1f34"). InnerVolumeSpecName "kube-api-access-7g4g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:48.833057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.833034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util" pod "ef6f6903-6471-4f86-b283-c154510d1f34" (UID: "ef6f6903-6471-4f86-b283-c154510d1f34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:48.929864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.929801 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4g8\" (UniqueName: \"kubernetes.io/projected/ef6f6903-6471-4f86-b283-c154510d1f34-kube-api-access-7g4g8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:48.929864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.929825 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:48.929864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:48.929836 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6f6903-6471-4f86-b283-c154510d1f34-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.604553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:49.604516 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" event={"ID":"ef6f6903-6471-4f86-b283-c154510d1f34","Type":"ContainerDied","Data":"485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609"} Oct 30 19:10:49.604553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:49.604552 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jhdvt" Oct 30 19:10:49.604748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:49.604566 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485a3b5842cd5e189db64209363edd325bcfa11508fc36459b0ca8a3f099a609" Oct 30 19:10:51.756489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:51.756446 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:52.006654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006579 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-gldn5"] Oct 30 19:10:52.006865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006854 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="util" Oct 30 19:10:52.006906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006867 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="util" Oct 30 19:10:52.006906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006879 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="pull" Oct 30 19:10:52.006906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006884 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="pull" Oct 30 19:10:52.006906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006903 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="extract" Oct 30 19:10:52.007016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006908 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="extract" Oct 30 19:10:52.007016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.006945 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6f6903-6471-4f86-b283-c154510d1f34" containerName="extract" Oct 30 19:10:52.013655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.013631 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.028323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.028298 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-gldn5"] Oct 30 19:10:52.037249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.037227 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-76d6g\"" Oct 30 19:10:52.155736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.155699 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-bound-sa-token\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.155878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.155754 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrxx\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-kube-api-access-gfrxx\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.256920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.256823 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-bound-sa-token\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.256920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.256891 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrxx\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-kube-api-access-gfrxx\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.273960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.273932 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-bound-sa-token\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.274066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.273937 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrxx\" (UniqueName: \"kubernetes.io/projected/396bb04c-d1d2-4909-adfb-1ae1b7844a94-kube-api-access-gfrxx\") pod \"cert-manager-597b95cb84-gldn5\" (UID: \"396bb04c-d1d2-4909-adfb-1ae1b7844a94\") " pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.325444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.325416 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-gldn5" Oct 30 19:10:52.616075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:52.616038 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-gldn5"] Oct 30 19:10:52.620733 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:10:52.620710 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod396bb04c_d1d2_4909_adfb_1ae1b7844a94.slice/crio-2300f7f1562c77d4edc6246c2ab4360e2ad2e1ba95ff0d03f8f913628945f010 WatchSource:0}: Error finding container 2300f7f1562c77d4edc6246c2ab4360e2ad2e1ba95ff0d03f8f913628945f010: Status 404 returned error can't find the container with id 2300f7f1562c77d4edc6246c2ab4360e2ad2e1ba95ff0d03f8f913628945f010 Oct 30 19:10:53.333083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.333043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.465583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.465542 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.600941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.600867 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" Oct 30 19:10:53.620979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.620944 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-gldn5" event={"ID":"396bb04c-d1d2-4909-adfb-1ae1b7844a94","Type":"ContainerStarted","Data":"54e9cefe41b5cdf6657ef7497b58a588a3ed554e22c15fa96fc2d79cce8d6d08"} Oct 30 19:10:53.621139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.620989 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-gldn5" event={"ID":"396bb04c-d1d2-4909-adfb-1ae1b7844a94","Type":"ContainerStarted","Data":"2300f7f1562c77d4edc6246c2ab4360e2ad2e1ba95ff0d03f8f913628945f010"} Oct 30 19:10:53.668273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:53.668224 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-gldn5" podStartSLOduration=2.668210546 podStartE2EDuration="2.668210546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:53.667011729 +0000 UTC m=+361.378537098" watchObservedRunningTime="2026-10-30 19:10:53.668210546 +0000 UTC m=+361.379735912" Oct 30 19:10:56.522520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:10:56.522480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:00.884650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:00.884598 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:01.047712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:01.047680 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:02.493459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:02.493423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:03.920976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.920944 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw"] Oct 30 19:11:03.925143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.925122 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:03.947340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.947310 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jbhvp\"" Oct 30 19:11:03.947340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.947337 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:11:03.947533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.947350 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:11:03.950171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:03.950143 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw"] Oct 30 19:11:04.046146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.046086 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.046347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.046211 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.046347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.046273 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqrw\" (UniqueName: \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.147240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.147204 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqrw\" (UniqueName: \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.147405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.147274 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.147405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.147307 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.147639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.147618 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.147718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.147677 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.164790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.164759 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqrw\" (UniqueName: \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.237224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.237159 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:04.526041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.525956 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw"] Oct 30 19:11:04.530785 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:11:04.530758 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfce8163_e615_44d1_b023_336428adef01.slice/crio-455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e WatchSource:0}: Error finding container 455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e: Status 404 returned error can't find the container with id 455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e Oct 30 19:11:04.664345 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.664314 4990 generic.go:356] "Generic (PLEG): container finished" podID="bfce8163-e615-44d1-b023-336428adef01" containerID="0bd51918abd6e6c599392aa2296fae46070c767ee3bf776f16d67aa377042021" exitCode=0 Oct 30 19:11:04.664480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.664385 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" event={"ID":"bfce8163-e615-44d1-b023-336428adef01","Type":"ContainerDied","Data":"0bd51918abd6e6c599392aa2296fae46070c767ee3bf776f16d67aa377042021"} Oct 30 19:11:04.664480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:04.664438 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" event={"ID":"bfce8163-e615-44d1-b023-336428adef01","Type":"ContainerStarted","Data":"455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e"} Oct 30 19:11:05.096109 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:11:05.096066 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:11:05.670028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:05.669993 4990 generic.go:356] "Generic (PLEG): container finished" podID="bfce8163-e615-44d1-b023-336428adef01" containerID="ab8d50ae1430c6d36d508761467b52ff61757ff1969f0dfbecf7022b9373add5" exitCode=0 Oct 30 19:11:05.670238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:05.670068 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" event={"ID":"bfce8163-e615-44d1-b023-336428adef01","Type":"ContainerDied","Data":"ab8d50ae1430c6d36d508761467b52ff61757ff1969f0dfbecf7022b9373add5"} Oct 30 19:11:06.676067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:06.676036 4990 generic.go:356] "Generic (PLEG): container finished" podID="bfce8163-e615-44d1-b023-336428adef01" containerID="53d83684d34feaa744a1da48c99187b3b446cfa3d6ed2aa6dc732a69cb757d41" exitCode=0 Oct 30 19:11:06.676474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:06.676085 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" event={"ID":"bfce8163-e615-44d1-b023-336428adef01","Type":"ContainerDied","Data":"53d83684d34feaa744a1da48c99187b3b446cfa3d6ed2aa6dc732a69cb757d41"} Oct 30 19:11:07.795976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.795953 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:07.973224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.973156 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\" (UniqueName: \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\") pod \"bfce8163-e615-44d1-b023-336428adef01\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " Oct 30 19:11:07.973347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.973251 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\") pod \"bfce8163-e615-44d1-b023-336428adef01\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " Oct 30 19:11:07.973347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.973276 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\") pod \"bfce8163-e615-44d1-b023-336428adef01\" (UID: \"bfce8163-e615-44d1-b023-336428adef01\") " Oct 30 19:11:07.973788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.973768 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle" pod "bfce8163-e615-44d1-b023-336428adef01" (UID: "bfce8163-e615-44d1-b023-336428adef01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:07.975199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.975167 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw" pod "bfce8163-e615-44d1-b023-336428adef01" (UID: "bfce8163-e615-44d1-b023-336428adef01"). InnerVolumeSpecName "kube-api-access-vkqrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:07.978594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:07.978577 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util" pod "bfce8163-e615-44d1-b023-336428adef01" (UID: "bfce8163-e615-44d1-b023-336428adef01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:08.073704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.073676 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:11:08.073704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.073701 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfce8163-e615-44d1-b023-336428adef01-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:11:08.073704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.073710 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqrw\" (UniqueName: \"kubernetes.io/projected/bfce8163-e615-44d1-b023-336428adef01-kube-api-access-vkqrw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:11:08.686083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.686005 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" event={"ID":"bfce8163-e615-44d1-b023-336428adef01","Type":"ContainerDied","Data":"455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e"} Oct 30 19:11:08.686083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.686023 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p5bkw" Oct 30 19:11:08.686083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:08.686054 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="455003080a9bad9a48d5a40451517dffc7e578d9181f730de0eb7d4acc32a51e" Oct 30 19:11:13.537069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:13.537024 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:13.778219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:13.778175 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.137542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.137500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.232218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232189 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt"] Oct 30 19:11:14.232462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232450 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="extract" Oct 30 19:11:14.232512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232463 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="extract" Oct 30 19:11:14.232512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232477 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="util" Oct 30 19:11:14.232512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232482 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="util" Oct 30 19:11:14.232512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232500 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="pull" Oct 30 19:11:14.232512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232505 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="pull" Oct 30 19:11:14.232672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.232542 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfce8163-e615-44d1-b023-336428adef01" containerName="extract" Oct 30 19:11:14.234545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.234529 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.259682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.259661 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:11:14.264781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.264762 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:11:14.264889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.264801 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:11:14.264889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.264815 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:11:14.264889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.264816 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:11:14.264889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.264832 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2lq27\"" Oct 30 19:11:14.328854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.328825 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt"] Oct 30 19:11:14.390127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.390040 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.415807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.415779 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.415936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.415817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.415936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.415845 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.415936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.415907 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.416061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.415985 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxf8\" (UniqueName: \"kubernetes.io/projected/7bd08d52-d04c-4956-b88f-c03db1cf376c-kube-api-access-tjxf8\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.416061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.416008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517065 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517130 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517157 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517210 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxf8\" (UniqueName: \"kubernetes.io/projected/7bd08d52-d04c-4956-b88f-c03db1cf376c-kube-api-access-tjxf8\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517236 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517287 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.517645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.517616 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.519362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.519334 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd08d52-d04c-4956-b88f-c03db1cf376c-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.519485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.519443 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.519653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.519633 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.519716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.519665 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bd08d52-d04c-4956-b88f-c03db1cf376c-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.534519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.534499 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxf8\" (UniqueName: \"kubernetes.io/projected/7bd08d52-d04c-4956-b88f-c03db1cf376c-kube-api-access-tjxf8\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ldgdt\" (UID: \"7bd08d52-d04c-4956-b88f-c03db1cf376c\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.546295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.546279 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:14.634200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.634163 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.794526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.794449 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.892446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.892415 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt"] Oct 30 19:11:14.895606 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:11:14.895580 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd08d52_d04c_4956_b88f_c03db1cf376c.slice/crio-508c533a7fb3c584f813e1c7c479cca23228ed1fa2065e44071ace9ca19da833 WatchSource:0}: Error finding container 508c533a7fb3c584f813e1c7c479cca23228ed1fa2065e44071ace9ca19da833: Status 404 returned error can't find the container with id 508c533a7fb3c584f813e1c7c479cca23228ed1fa2065e44071ace9ca19da833 Oct 30 19:11:14.946368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:14.946337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:15.713260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:15.713220 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" event={"ID":"7bd08d52-d04c-4956-b88f-c03db1cf376c","Type":"ContainerStarted","Data":"508c533a7fb3c584f813e1c7c479cca23228ed1fa2065e44071ace9ca19da833"} Oct 30 19:11:18.728836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:18.728801 4990 generic.go:356] "Generic (PLEG): container finished" podID="7bd08d52-d04c-4956-b88f-c03db1cf376c" containerID="a8f5338b51aef4f52a5c2c64a95bd3bf5277ffc35f8117aa44de2af297a38f0d" exitCode=0 Oct 30 19:11:18.729246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:18.728890 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" event={"ID":"7bd08d52-d04c-4956-b88f-c03db1cf376c","Type":"ContainerDied","Data":"a8f5338b51aef4f52a5c2c64a95bd3bf5277ffc35f8117aa44de2af297a38f0d"} Oct 30 19:11:19.735524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:19.735485 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" event={"ID":"7bd08d52-d04c-4956-b88f-c03db1cf376c","Type":"ContainerStarted","Data":"12ae4325da25ec855910fdb97adfcc230fa8f5dbb9f58389e4d86528de0db2de"} Oct 30 19:11:19.736006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:19.735621 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:19.808692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:19.808644 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" podStartSLOduration=2.206118824 podStartE2EDuration="5.808631276s" totalImagesPullingTime="3.198528604s" totalInitContainerRuntime="403.983848ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:19.798554 +0000 UTC m=+387.510079368" watchObservedRunningTime="2026-10-30 19:11:19.808631276 +0000 UTC m=+387.520156640" Oct 30 19:11:20.425274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:20.425245 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:20.569358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:20.569325 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.742138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:30.742109 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ldgdt" Oct 30 19:11:31.473210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.473170 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.627063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.627031 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59"] Oct 30 19:11:31.631818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.631796 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.655515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.653971 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 19:11:31.660709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.660532 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:11:31.663478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.663453 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 19:11:31.663667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.663651 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-svszl\"" Oct 30 19:11:31.663751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.663743 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 19:11:31.663864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.663846 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 19:11:31.665414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.665395 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59"] Oct 30 19:11:31.668890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.668866 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.751885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.751814 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eafed21c-629e-4efd-a312-aabcb81f5cd4-manager-config\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.751885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.751874 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdv8\" (UniqueName: \"kubernetes.io/projected/eafed21c-629e-4efd-a312-aabcb81f5cd4-kube-api-access-jgdv8\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.752302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.751967 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.752302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.752020 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.852727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.852698 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eafed21c-629e-4efd-a312-aabcb81f5cd4-manager-config\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.852893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.852750 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdv8\" (UniqueName: \"kubernetes.io/projected/eafed21c-629e-4efd-a312-aabcb81f5cd4-kube-api-access-jgdv8\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.852893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.852771 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.852893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.852803 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.853456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.853428 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eafed21c-629e-4efd-a312-aabcb81f5cd4-manager-config\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.855288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.855268 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.855375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.855339 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eafed21c-629e-4efd-a312-aabcb81f5cd4-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.870416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.870389 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdv8\" (UniqueName: \"kubernetes.io/projected/eafed21c-629e-4efd-a312-aabcb81f5cd4-kube-api-access-jgdv8\") pod \"lws-controller-manager-5c54b56ff8-4bj59\" (UID: \"eafed21c-629e-4efd-a312-aabcb81f5cd4\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:31.942387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:31.942363 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:32.243240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:32.243208 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59"] Oct 30 19:11:32.247792 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:11:32.247755 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafed21c_629e_4efd_a312_aabcb81f5cd4.slice/crio-20c7166ef00f96dd3eb2e728291fdd806ab66dccdea36bf377bfdc1dd903a0ba WatchSource:0}: Error finding container 20c7166ef00f96dd3eb2e728291fdd806ab66dccdea36bf377bfdc1dd903a0ba: Status 404 returned error can't find the container with id 20c7166ef00f96dd3eb2e728291fdd806ab66dccdea36bf377bfdc1dd903a0ba Oct 30 19:11:32.788301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:32.788262 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" event={"ID":"eafed21c-629e-4efd-a312-aabcb81f5cd4","Type":"ContainerStarted","Data":"20c7166ef00f96dd3eb2e728291fdd806ab66dccdea36bf377bfdc1dd903a0ba"} Oct 30 19:11:33.765324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:33.765286 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:33.935425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:33.935387 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:34.798044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:34.798012 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" event={"ID":"eafed21c-629e-4efd-a312-aabcb81f5cd4","Type":"ContainerStarted","Data":"84acc9a96704a4571323b775142402faefd964cce9ca0f1ddb7965b39d233e4c"} Oct 30 19:11:34.798241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:34.798071 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:34.832833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:34.832784 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" podStartSLOduration=2.292278551 podStartE2EDuration="3.832771302s" totalImagesPullingTime="1.540492751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:34.831678503 +0000 UTC m=+402.543203868" watchObservedRunningTime="2026-10-30 19:11:34.832771302 +0000 UTC m=+402.544296665" Oct 30 19:11:35.093988 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:11:35.093955 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:11:37.078628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:37.078592 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.383806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:37.383725 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.706453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:37.706360 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:45.217431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.217395 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g"] Oct 30 19:11:45.221909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.221888 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.235739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.235716 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:11:45.235739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.235733 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:11:45.247479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.247452 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g"] Oct 30 19:11:45.259305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.259275 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:11:45.259305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.259302 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wmzc4\"" Oct 30 19:11:45.266317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.266290 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:11:45.266462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.266328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:45.359349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.359317 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7tr\" (UniqueName: \"kubernetes.io/projected/83a243fd-8ba3-4e72-9760-6583b0458d47-kube-api-access-tm7tr\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.359524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.359388 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a243fd-8ba3-4e72-9760-6583b0458d47-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.359524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.359478 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a243fd-8ba3-4e72-9760-6583b0458d47-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.415649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.415613 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:45.460152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.460114 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7tr\" (UniqueName: \"kubernetes.io/projected/83a243fd-8ba3-4e72-9760-6583b0458d47-kube-api-access-tm7tr\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.460364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.460187 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a243fd-8ba3-4e72-9760-6583b0458d47-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.460364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.460215 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a243fd-8ba3-4e72-9760-6583b0458d47-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.462598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.462568 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a243fd-8ba3-4e72-9760-6583b0458d47-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.462708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.462678 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a243fd-8ba3-4e72-9760-6583b0458d47-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.479890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.479818 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7tr\" (UniqueName: \"kubernetes.io/projected/83a243fd-8ba3-4e72-9760-6583b0458d47-kube-api-access-tm7tr\") pod \"kube-auth-proxy-6c7d47d6c9-dbd6g\" (UID: \"83a243fd-8ba3-4e72-9760-6583b0458d47\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.534758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.534712 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" Oct 30 19:11:45.804540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.804512 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-4bj59" Oct 30 19:11:45.848219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:45.848191 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g"] Oct 30 19:11:45.852056 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:11:45.852027 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a243fd_8ba3_4e72_9760_6583b0458d47.slice/crio-f629f782c501c86f5af8bd58e44ccfecbb16f8230fa5eb7e2b341fb26973ff68 WatchSource:0}: Error finding container f629f782c501c86f5af8bd58e44ccfecbb16f8230fa5eb7e2b341fb26973ff68: Status 404 returned error can't find the container with id f629f782c501c86f5af8bd58e44ccfecbb16f8230fa5eb7e2b341fb26973ff68 Oct 30 19:11:46.683921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:46.683819 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:46.853065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:46.853027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:46.855884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:46.855853 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" event={"ID":"83a243fd-8ba3-4e72-9760-6583b0458d47","Type":"ContainerStarted","Data":"f629f782c501c86f5af8bd58e44ccfecbb16f8230fa5eb7e2b341fb26973ff68"} Oct 30 19:11:47.724269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:47.724229 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:48.865853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:48.865814 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" event={"ID":"83a243fd-8ba3-4e72-9760-6583b0458d47","Type":"ContainerStarted","Data":"af2ead6cdd754d3ae3f44e70a1f04fa63274b8cf3051d6bccfc8d33f87f2b4a1"} Oct 30 19:11:48.904082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:48.904027 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-dbd6g" podStartSLOduration=1.529458736 podStartE2EDuration="3.904013258s" totalImagesPullingTime="2.374554522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:48.902556105 +0000 UTC m=+416.614081470" watchObservedRunningTime="2026-10-30 19:11:48.904013258 +0000 UTC m=+416.615538615" Oct 30 19:11:49.795226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:49.795193 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:49.957237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:49.957197 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:50.493358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:50.493327 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:50.641490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:50.641450 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:51.139265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:51.139221 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:51.280696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:51.280659 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:51.868668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:51.868630 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:56.622317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:11:56.622285 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:05.095839 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:12:05.095801 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:12:35.096122 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:12:35.096020 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:12:37.039852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:37.039820 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:37.200170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:37.200129 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:37.379643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:37.379606 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:37.530254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:37.530203 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:37.831140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:37.831110 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:39.615264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:39.615225 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:39.794620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:39.794591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:39.955341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:39.955260 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:40.095823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:40.095792 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:50.824253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:50.824216 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:51.604407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:51.604366 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:56.747724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:12:56.747684 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:05.094892 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:13:05.094854 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:13:21.275974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:13:21.275935 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:35.094536 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:13:35.094503 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:13:53.502350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:13:53.502307 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:53.649390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:13:53.649357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:56.877753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:13:56.877718 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:05.095398 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:14:05.095310 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:14:07.377479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:07.377442 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.247859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:09.247828 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:23.446787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:23.446746 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:29.531417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.531374 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc"] Oct 30 19:14:29.535282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.535261 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.558812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.558767 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:14:29.559001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.558824 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cqvkq\"" Oct 30 19:14:29.559001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.558961 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:14:29.560678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.560658 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc"] Oct 30 19:14:29.585511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.585470 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.585511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.585514 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.585745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.585540 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx4j\" (UniqueName: \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.601566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.601530 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg"] Oct 30 19:14:29.605149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.605124 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.617288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.617262 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bk4cl\"" Oct 30 19:14:29.626004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.625977 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg"] Oct 30 19:14:29.686418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686366 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.686418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686428 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.686666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686457 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.686666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx4j\" (UniqueName: \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.686666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686504 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fkb\" (UniqueName: \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.686826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686749 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.686826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686755 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.687001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.686979 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.697838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.697803 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448"] Oct 30 19:14:29.701637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.701609 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.704319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.704288 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx4j\" (UniqueName: \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.713780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.713755 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2v9ls\"" Oct 30 19:14:29.721894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.721866 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448"] Oct 30 19:14:29.788344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788263 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.788344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788325 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8795h\" (UniqueName: \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.788565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788388 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.788565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788539 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.788677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788587 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.788677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788608 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.788677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788636 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fkb\" (UniqueName: \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.788894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.788869 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.806705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.806672 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx"] Oct 30 19:14:29.810514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.810484 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.810951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.810923 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fkb\" (UniqueName: \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.824146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.824124 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fc67w\"" Oct 30 19:14:29.830650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.830621 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx"] Oct 30 19:14:29.850225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.850194 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:29.889626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889549 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.889626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889612 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.889889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889643 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.889889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889681 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzqw\" (UniqueName: \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.889889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889737 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8795h\" (UniqueName: \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.889889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.889766 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.890049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.890006 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.890108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.890041 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.908121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.907959 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8795h\" (UniqueName: \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:29.920888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.920609 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:29.990854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.990759 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.990854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.990830 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzqw\" (UniqueName: \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.991231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.991205 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.991583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.991565 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:29.991660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:29.991638 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:30.009172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.009136 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzqw\" (UniqueName: \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:30.023966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.023940 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:30.131677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.131639 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:30.148735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.148700 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc"] Oct 30 19:14:30.152880 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:30.152853 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f819ec_142e_4cfe_b720_06d46ed1be6c.slice/crio-485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a WatchSource:0}: Error finding container 485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a: Status 404 returned error can't find the container with id 485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a Oct 30 19:14:30.217329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.217289 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg"] Oct 30 19:14:30.220952 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:30.220917 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01adb5b_ba35_458a_988a_0afa6aec781f.slice/crio-46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547 WatchSource:0}: Error finding container 46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547: Status 404 returned error can't find the container with id 46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547 Oct 30 19:14:30.319871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.319836 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448"] Oct 30 19:14:30.330715 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:30.330681 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b7f3cb_d689_495b_9804_27ad1ee685b9.slice/crio-faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c WatchSource:0}: Error finding container faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c: Status 404 returned error can't find the container with id faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c Oct 30 19:14:30.430854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.430774 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx"] Oct 30 19:14:30.493054 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:30.493015 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a66fcd_767f_48e8_81c4_5ffa8c7dee8b.slice/crio-9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063 WatchSource:0}: Error finding container 9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063: Status 404 returned error can't find the container with id 9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063 Oct 30 19:14:30.519691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.519656 4990 generic.go:356] "Generic (PLEG): container finished" podID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerID="379dce2d5f5ada2f2ef848019f4f4697b805814cee9becc5a10fd151bd0ec3e2" exitCode=0 Oct 30 19:14:30.519863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.519742 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" event={"ID":"a3b7f3cb-d689-495b-9804-27ad1ee685b9","Type":"ContainerDied","Data":"379dce2d5f5ada2f2ef848019f4f4697b805814cee9becc5a10fd151bd0ec3e2"} Oct 30 19:14:30.519863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.519790 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" event={"ID":"a3b7f3cb-d689-495b-9804-27ad1ee685b9","Type":"ContainerStarted","Data":"faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c"} Oct 30 19:14:30.521236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.521209 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" event={"ID":"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b","Type":"ContainerStarted","Data":"9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063"} Oct 30 19:14:30.522913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.522891 4990 generic.go:356] "Generic (PLEG): container finished" podID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerID="d3e40ca6f19e22b56824a68360a2bde2eca5bc3da78cfac1d217c4faa5a5e5ec" exitCode=0 Oct 30 19:14:30.522997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.522954 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" event={"ID":"d01adb5b-ba35-458a-988a-0afa6aec781f","Type":"ContainerDied","Data":"d3e40ca6f19e22b56824a68360a2bde2eca5bc3da78cfac1d217c4faa5a5e5ec"} Oct 30 19:14:30.522997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.522984 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" event={"ID":"d01adb5b-ba35-458a-988a-0afa6aec781f","Type":"ContainerStarted","Data":"46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547"} Oct 30 19:14:30.524694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.524675 4990 generic.go:356] "Generic (PLEG): container finished" podID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerID="6a6de955fa13bd545bb6dfd66b6d5d528b19965be0912bc7f6a2a3253f2ff5a9" exitCode=0 Oct 30 19:14:30.524788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.524709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" event={"ID":"91f819ec-142e-4cfe-b720-06d46ed1be6c","Type":"ContainerDied","Data":"6a6de955fa13bd545bb6dfd66b6d5d528b19965be0912bc7f6a2a3253f2ff5a9"} Oct 30 19:14:30.524788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:30.524743 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" event={"ID":"91f819ec-142e-4cfe-b720-06d46ed1be6c","Type":"ContainerStarted","Data":"485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a"} Oct 30 19:14:31.026333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.026258 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:31.530567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.530479 4990 generic.go:356] "Generic (PLEG): container finished" podID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerID="a7095fbe6fb10c6c77067054c241ac72371515010767e0990ef558e926d6e4aa" exitCode=0 Oct 30 19:14:31.530743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.530570 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" event={"ID":"91f819ec-142e-4cfe-b720-06d46ed1be6c","Type":"ContainerDied","Data":"a7095fbe6fb10c6c77067054c241ac72371515010767e0990ef558e926d6e4aa"} Oct 30 19:14:31.532467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.532443 4990 generic.go:356] "Generic (PLEG): container finished" podID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerID="727f747f95cfbad52d59dc6670f5cc9dd1289972ce5c17d3f059cd0cb3be4680" exitCode=0 Oct 30 19:14:31.532555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.532532 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" event={"ID":"a3b7f3cb-d689-495b-9804-27ad1ee685b9","Type":"ContainerDied","Data":"727f747f95cfbad52d59dc6670f5cc9dd1289972ce5c17d3f059cd0cb3be4680"} Oct 30 19:14:31.534205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.534182 4990 generic.go:356] "Generic (PLEG): container finished" podID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerID="ae89b7f6143b7d3c7147394d9447633231f45e887fde247ab6ccc2fa3bdd32c7" exitCode=0 Oct 30 19:14:31.534285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:31.534237 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" event={"ID":"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b","Type":"ContainerDied","Data":"ae89b7f6143b7d3c7147394d9447633231f45e887fde247ab6ccc2fa3bdd32c7"} Oct 30 19:14:32.541242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:32.541198 4990 generic.go:356] "Generic (PLEG): container finished" podID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerID="61c7561defbaab50afc068ded29675d89991f3e0f71aaab2381d749edb39d9f0" exitCode=0 Oct 30 19:14:32.541677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:32.541269 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" event={"ID":"91f819ec-142e-4cfe-b720-06d46ed1be6c","Type":"ContainerDied","Data":"61c7561defbaab50afc068ded29675d89991f3e0f71aaab2381d749edb39d9f0"} Oct 30 19:14:32.543312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:32.543288 4990 generic.go:356] "Generic (PLEG): container finished" podID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerID="de40676e80a25e4aaa11db5b5b3a14c5675fc68e89dcb1aaf0f25783e483ae48" exitCode=0 Oct 30 19:14:32.543445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:32.543373 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" event={"ID":"a3b7f3cb-d689-495b-9804-27ad1ee685b9","Type":"ContainerDied","Data":"de40676e80a25e4aaa11db5b5b3a14c5675fc68e89dcb1aaf0f25783e483ae48"} Oct 30 19:14:33.700043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.700010 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:33.702826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.702804 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:33.819001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.818962 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\" (UniqueName: \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\") pod \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " Oct 30 19:14:33.819187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819012 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\" (UniqueName: \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\") pod \"91f819ec-142e-4cfe-b720-06d46ed1be6c\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " Oct 30 19:14:33.819187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\") pod \"91f819ec-142e-4cfe-b720-06d46ed1be6c\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " Oct 30 19:14:33.819187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819108 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\") pod \"91f819ec-142e-4cfe-b720-06d46ed1be6c\" (UID: \"91f819ec-142e-4cfe-b720-06d46ed1be6c\") " Oct 30 19:14:33.819187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819137 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\") pod \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " Oct 30 19:14:33.819187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819174 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\") pod \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\" (UID: \"a3b7f3cb-d689-495b-9804-27ad1ee685b9\") " Oct 30 19:14:33.819767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819735 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle" pod "91f819ec-142e-4cfe-b720-06d46ed1be6c" (UID: "91f819ec-142e-4cfe-b720-06d46ed1be6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:33.819899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.819820 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle" pod "a3b7f3cb-d689-495b-9804-27ad1ee685b9" (UID: "a3b7f3cb-d689-495b-9804-27ad1ee685b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:33.821289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.821259 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j" pod "91f819ec-142e-4cfe-b720-06d46ed1be6c" (UID: "91f819ec-142e-4cfe-b720-06d46ed1be6c"). InnerVolumeSpecName "kube-api-access-5rx4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:33.821383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.821307 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h" pod "a3b7f3cb-d689-495b-9804-27ad1ee685b9" (UID: "a3b7f3cb-d689-495b-9804-27ad1ee685b9"). InnerVolumeSpecName "kube-api-access-8795h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:33.824330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.824281 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util" pod "91f819ec-142e-4cfe-b720-06d46ed1be6c" (UID: "91f819ec-142e-4cfe-b720-06d46ed1be6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:33.827377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.827323 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util" pod "a3b7f3cb-d689-495b-9804-27ad1ee685b9" (UID: "a3b7f3cb-d689-495b-9804-27ad1ee685b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:33.920421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920390 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.920421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920415 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8795h\" (UniqueName: \"kubernetes.io/projected/a3b7f3cb-d689-495b-9804-27ad1ee685b9-kube-api-access-8795h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.920421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920425 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rx4j\" (UniqueName: \"kubernetes.io/projected/91f819ec-142e-4cfe-b720-06d46ed1be6c-kube-api-access-5rx4j\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.920631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920456 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.920631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920465 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91f819ec-142e-4cfe-b720-06d46ed1be6c-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:33.920631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:33.920473 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3b7f3cb-d689-495b-9804-27ad1ee685b9-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:34.558151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.558114 4990 generic.go:356] "Generic (PLEG): container finished" podID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerID="eb34dbeb3020ceb10a0a43f29d9861c83bd40e965fc16f9a9e7f44c44aa0336f" exitCode=0 Oct 30 19:14:34.558365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.558192 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" event={"ID":"d01adb5b-ba35-458a-988a-0afa6aec781f","Type":"ContainerDied","Data":"eb34dbeb3020ceb10a0a43f29d9861c83bd40e965fc16f9a9e7f44c44aa0336f"} Oct 30 19:14:34.560399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.560364 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" event={"ID":"91f819ec-142e-4cfe-b720-06d46ed1be6c","Type":"ContainerDied","Data":"485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a"} Oct 30 19:14:34.560478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.560413 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485217ce1d917ad0ccef9cdbe43d8155f7ba8c9a8b0d36cb094871795d22553a" Oct 30 19:14:34.560478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.560379 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgpcxc" Oct 30 19:14:34.562378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.562349 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" Oct 30 19:14:34.562525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.562507 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qc448" event={"ID":"a3b7f3cb-d689-495b-9804-27ad1ee685b9","Type":"ContainerDied","Data":"faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c"} Oct 30 19:14:34.562627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.562613 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa2ce4b18878251c24b9b5568c86a244aa715107a81b1ccc63fd2c81683a96c" Oct 30 19:14:34.564277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.564258 4990 generic.go:356] "Generic (PLEG): container finished" podID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerID="d3947285900c2d0f11dad4954ebaaeaa9a5bbfdfa0a715e57d74fa92c3cc18cd" exitCode=0 Oct 30 19:14:34.564360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:34.564283 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" event={"ID":"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b","Type":"ContainerDied","Data":"d3947285900c2d0f11dad4954ebaaeaa9a5bbfdfa0a715e57d74fa92c3cc18cd"} Oct 30 19:14:35.095051 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:14:35.095015 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:14:35.574611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:35.574579 4990 generic.go:356] "Generic (PLEG): container finished" podID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerID="5b77316d2d1d7d15c1d223d579f990aae51978004d5e14f023ce4bc848db3ac9" exitCode=0 Oct 30 19:14:35.574824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:35.574666 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" event={"ID":"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b","Type":"ContainerDied","Data":"5b77316d2d1d7d15c1d223d579f990aae51978004d5e14f023ce4bc848db3ac9"} Oct 30 19:14:35.576737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:35.576709 4990 generic.go:356] "Generic (PLEG): container finished" podID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerID="9d0e25c77811fdeef15cc9fd13a705655dea405437d179ec99ad0b43ce5b3c18" exitCode=0 Oct 30 19:14:35.576865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:35.576788 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" event={"ID":"d01adb5b-ba35-458a-988a-0afa6aec781f","Type":"ContainerDied","Data":"9d0e25c77811fdeef15cc9fd13a705655dea405437d179ec99ad0b43ce5b3c18"} Oct 30 19:14:36.725306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.725276 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:36.728286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.728260 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:36.849076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.848978 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\") pod \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " Oct 30 19:14:36.849076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849025 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\" (UniqueName: \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\") pod \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " Oct 30 19:14:36.849076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849048 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\" (UniqueName: \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\") pod \"d01adb5b-ba35-458a-988a-0afa6aec781f\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " Oct 30 19:14:36.849076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849072 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\") pod \"d01adb5b-ba35-458a-988a-0afa6aec781f\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " Oct 30 19:14:36.849416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849113 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\") pod \"d01adb5b-ba35-458a-988a-0afa6aec781f\" (UID: \"d01adb5b-ba35-458a-988a-0afa6aec781f\") " Oct 30 19:14:36.849416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849200 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\") pod \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\" (UID: \"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b\") " Oct 30 19:14:36.849657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849628 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle" pod "d01adb5b-ba35-458a-988a-0afa6aec781f" (UID: "d01adb5b-ba35-458a-988a-0afa6aec781f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:36.849934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.849914 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle" pod "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" (UID: "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:36.851702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.851673 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb" pod "d01adb5b-ba35-458a-988a-0afa6aec781f" (UID: "d01adb5b-ba35-458a-988a-0afa6aec781f"). InnerVolumeSpecName "kube-api-access-77fkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:36.851802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.851726 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw" pod "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" (UID: "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b"). InnerVolumeSpecName "kube-api-access-fjzqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:36.854753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.854712 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util" pod "d01adb5b-ba35-458a-988a-0afa6aec781f" (UID: "d01adb5b-ba35-458a-988a-0afa6aec781f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:36.854975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.854952 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util" pod "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" (UID: "97a66fcd-767f-48e8-81c4-5ffa8c7dee8b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:36.950299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950255 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:36.950299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950292 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:36.950299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950304 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzqw\" (UniqueName: \"kubernetes.io/projected/97a66fcd-767f-48e8-81c4-5ffa8c7dee8b-kube-api-access-fjzqw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:36.950539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950319 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fkb\" (UniqueName: \"kubernetes.io/projected/d01adb5b-ba35-458a-988a-0afa6aec781f-kube-api-access-77fkb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:36.950539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950332 4990 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-bundle\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:36.950539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:36.950343 4990 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d01adb5b-ba35-458a-988a-0afa6aec781f-util\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:14:37.588414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.588386 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" Oct 30 19:14:37.588414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.588399 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alhbnx" event={"ID":"97a66fcd-767f-48e8-81c4-5ffa8c7dee8b","Type":"ContainerDied","Data":"9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063"} Oct 30 19:14:37.588665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.588446 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9f33e5dfdb7083b00f719aac9838c0016b63713b50bb16da53bf0301490063" Oct 30 19:14:37.590487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.590459 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" event={"ID":"d01adb5b-ba35-458a-988a-0afa6aec781f","Type":"ContainerDied","Data":"46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547"} Oct 30 19:14:37.590618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.590495 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46092ca0110cf23780e534ef1cdd007cb66276050fea8bed49e25ef7ed2cf547" Oct 30 19:14:37.590618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:37.590512 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hhgbg" Oct 30 19:14:43.098681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:43.098644 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:43.907447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:43.907411 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.969743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:46.969702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:47.221756 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:47.221653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:49.161387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:49.161337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:49.343550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:49.343506 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:50.906350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906304 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb"] Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906649 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906666 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906680 4990 cpu_manager.go:414] "containerMap: removing container" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906687 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906703 4990 cpu_manager.go:414] "containerMap: removing container" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906710 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906724 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906730 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906741 4990 cpu_manager.go:414] "containerMap: removing container" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906747 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906757 4990 cpu_manager.go:414] "containerMap: removing container" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906763 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906772 4990 cpu_manager.go:414] "containerMap: removing container" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906778 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906791 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906797 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906809 4990 cpu_manager.go:414] "containerMap: removing container" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906817 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="util" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906825 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906832 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="pull" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906843 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906849 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="extract" Oct 30 19:14:50.906847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906859 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="pull" Oct 30 19:14:50.907587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906865 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="pull" Oct 30 19:14:50.907587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906938 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01adb5b-ba35-458a-988a-0afa6aec781f" containerName="extract" Oct 30 19:14:50.907587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906949 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a66fcd-767f-48e8-81c4-5ffa8c7dee8b" containerName="extract" Oct 30 19:14:50.907587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906957 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f819ec-142e-4cfe-b720-06d46ed1be6c" containerName="extract" Oct 30 19:14:50.907587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.906968 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b7f3cb-d689-495b-9804-27ad1ee685b9" containerName="extract" Oct 30 19:14:50.909585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.909565 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:50.922576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.922538 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:14:50.932510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.932488 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9ttc8\"" Oct 30 19:14:50.932781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.932760 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:14:50.937374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:50.937341 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb"] Oct 30 19:14:51.024423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.024395 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:14:51.067422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.067393 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdkf\" (UniqueName: \"kubernetes.io/projected/e7a72d51-4726-4114-96de-9ec15e1ab570-kube-api-access-9cdkf\") pod \"dns-operator-controller-manager-65bfd79598-xrlsb\" (UID: \"e7a72d51-4726-4114-96de-9ec15e1ab570\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:51.168272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.168167 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdkf\" (UniqueName: \"kubernetes.io/projected/e7a72d51-4726-4114-96de-9ec15e1ab570-kube-api-access-9cdkf\") pod \"dns-operator-controller-manager-65bfd79598-xrlsb\" (UID: \"e7a72d51-4726-4114-96de-9ec15e1ab570\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:51.190156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.190128 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdkf\" (UniqueName: \"kubernetes.io/projected/e7a72d51-4726-4114-96de-9ec15e1ab570-kube-api-access-9cdkf\") pod \"dns-operator-controller-manager-65bfd79598-xrlsb\" (UID: \"e7a72d51-4726-4114-96de-9ec15e1ab570\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:51.220851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.220803 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:51.303443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.303401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:51.520614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.520532 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb"] Oct 30 19:14:51.524260 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:51.524219 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a72d51_4726_4114_96de_9ec15e1ab570.slice/crio-1bb52aae7936802cd807b60f33914baf89d7456a824f1034966cf750992513d2 WatchSource:0}: Error finding container 1bb52aae7936802cd807b60f33914baf89d7456a824f1034966cf750992513d2: Status 404 returned error can't find the container with id 1bb52aae7936802cd807b60f33914baf89d7456a824f1034966cf750992513d2 Oct 30 19:14:51.545305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.545275 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:51.651690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.651654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" event={"ID":"e7a72d51-4726-4114-96de-9ec15e1ab570","Type":"ContainerStarted","Data":"1bb52aae7936802cd807b60f33914baf89d7456a824f1034966cf750992513d2"} Oct 30 19:14:51.755349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:51.755312 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:52.323414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.323371 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:52.449293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.449268 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:14:52.449512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.449494 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:14:52.465920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.465891 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:14:52.466149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.466106 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:14:52.793401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:52.793315 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.038816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:53.038782 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.822842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:53.822808 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:14:54.666971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:54.666928 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" event={"ID":"e7a72d51-4726-4114-96de-9ec15e1ab570","Type":"ContainerStarted","Data":"3e2502bd73f95fe38399e4d87548956a8ffdf60e6cffda6628b98abc5acd097d"} Oct 30 19:14:54.667224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:54.667114 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:14:54.705507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:54.705452 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" podStartSLOduration=2.424131004 podStartE2EDuration="4.705435041s" totalImagesPullingTime="2.281304037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:54.705208262 +0000 UTC m=+602.416733630" watchObservedRunningTime="2026-10-30 19:14:54.705435041 +0000 UTC m=+602.416960406" Oct 30 19:14:55.356169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.356131 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v"] Oct 30 19:14:55.359568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.359546 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.379575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.379539 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzmp6\"" Oct 30 19:14:55.390930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.390897 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v"] Oct 30 19:14:55.506569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.506531 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgppv\" (UniqueName: \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.506771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.506607 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.607906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.607812 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.607906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.607876 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgppv\" (UniqueName: \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.608318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.608294 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.624881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.624848 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgppv\" (UniqueName: \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\") pod \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:55.672124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:55.672078 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:14:56.007800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:56.007712 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v"] Oct 30 19:14:56.011831 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:14:56.011807 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdb5260_fe44_4fda_bc18_bcc956c2d6cd.slice/crio-dbb5f71c88a89b1f3cee65fdf2df8e1a5c235c6e22b42ed76c87736b7f440648 WatchSource:0}: Error finding container dbb5f71c88a89b1f3cee65fdf2df8e1a5c235c6e22b42ed76c87736b7f440648: Status 404 returned error can't find the container with id dbb5f71c88a89b1f3cee65fdf2df8e1a5c235c6e22b42ed76c87736b7f440648 Oct 30 19:14:56.677904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:56.677872 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" event={"ID":"afdb5260-fe44-4fda-bc18-bcc956c2d6cd","Type":"ContainerStarted","Data":"dbb5f71c88a89b1f3cee65fdf2df8e1a5c235c6e22b42ed76c87736b7f440648"} Oct 30 19:14:56.935549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:56.935474 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.087557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:57.087526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.564983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:57.564946 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.738202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:14:57.738164 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:00.699419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:00.699381 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" event={"ID":"afdb5260-fe44-4fda-bc18-bcc956c2d6cd","Type":"ContainerStarted","Data":"b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6"} Oct 30 19:15:00.699818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:00.699482 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:00.757362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:00.757304 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" podStartSLOduration=2.051519523 podStartE2EDuration="5.757291611s" totalImagesPullingTime="3.705772088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:00.746132591 +0000 UTC m=+608.457657957" watchObservedRunningTime="2026-10-30 19:15:00.757291611 +0000 UTC m=+608.468816977" Oct 30 19:15:00.944331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:00.944294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:01.105441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:01.105405 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:02.502857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.502819 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:02.666954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.666918 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:02.907760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.907720 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6"] Oct 30 19:15:02.912019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.911997 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:02.924949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.924928 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpcvt\"" Oct 30 19:15:02.924949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.924930 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 19:15:02.934961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.934935 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 19:15:02.936498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.936479 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6"] Oct 30 19:15:02.950697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:02.950667 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:03.072791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.072745 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdq8\" (UniqueName: \"kubernetes.io/projected/e3f626ef-bed4-4974-9ba4-7778435cee18-kube-api-access-fvdq8\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.072991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.072874 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3f626ef-bed4-4974-9ba4-7778435cee18-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.072991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.072921 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3f626ef-bed4-4974-9ba4-7778435cee18-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.173831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.173754 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdq8\" (UniqueName: \"kubernetes.io/projected/e3f626ef-bed4-4974-9ba4-7778435cee18-kube-api-access-fvdq8\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.173831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.173807 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3f626ef-bed4-4974-9ba4-7778435cee18-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.174006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.173846 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3f626ef-bed4-4974-9ba4-7778435cee18-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.174608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.174576 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e3f626ef-bed4-4974-9ba4-7778435cee18-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.176308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.176286 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3f626ef-bed4-4974-9ba4-7778435cee18-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.191026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.190996 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdq8\" (UniqueName: \"kubernetes.io/projected/e3f626ef-bed4-4974-9ba4-7778435cee18-kube-api-access-fvdq8\") pod \"kuadrant-console-plugin-66c4fb575d-jqxt6\" (UID: \"e3f626ef-bed4-4974-9ba4-7778435cee18\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.202606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.202583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:03.223415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.223391 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" Oct 30 19:15:03.516875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.516787 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6"] Oct 30 19:15:03.521688 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:15:03.521650 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f626ef_bed4_4974_9ba4_7778435cee18.slice/crio-5de7433bc9640df0a01fdd08769e2bbd3ad5380c19163c9da6ec85735f12933d WatchSource:0}: Error finding container 5de7433bc9640df0a01fdd08769e2bbd3ad5380c19163c9da6ec85735f12933d: Status 404 returned error can't find the container with id 5de7433bc9640df0a01fdd08769e2bbd3ad5380c19163c9da6ec85735f12933d Oct 30 19:15:03.714416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:03.714381 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" event={"ID":"e3f626ef-bed4-4974-9ba4-7778435cee18","Type":"ContainerStarted","Data":"5de7433bc9640df0a01fdd08769e2bbd3ad5380c19163c9da6ec85735f12933d"} Oct 30 19:15:05.105679 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:05.105638 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:15:05.674976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:05.674942 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xrlsb" Oct 30 19:15:08.037975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:08.037946 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:08.864288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:08.864251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:09.005081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:09.005052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:09.743882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:09.743849 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" event={"ID":"e3f626ef-bed4-4974-9ba4-7778435cee18","Type":"ContainerStarted","Data":"96b4974f876257682723120cd83054829bbfb5638788e88a88f835e48c8cc7a8"} Oct 30 19:15:09.778211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:09.778158 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jqxt6" podStartSLOduration=2.426445067 podStartE2EDuration="7.778143566s" totalImagesPullingTime="5.351698499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:09.777479516 +0000 UTC m=+617.489004882" watchObservedRunningTime="2026-10-30 19:15:09.778143566 +0000 UTC m=+617.489668928" Oct 30 19:15:11.707147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:11.707083 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:14.562430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:14.562400 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:14.733011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:14.732975 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.283024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.282960 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v"] Oct 30 19:15:15.283312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.283218 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" containerName="manager" containerID="cri-o://b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6" gracePeriod=2 Oct 30 19:15:15.284640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.284610 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v"] Oct 30 19:15:15.296864 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:15.296824 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:15.536518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.536452 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:15.547619 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:15.547583 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:15.578135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.578105 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\") pod \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " Oct 30 19:15:15.578526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.578153 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\" (UniqueName: \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\") pod \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\" (UID: \"afdb5260-fe44-4fda-bc18-bcc956c2d6cd\") " Oct 30 19:15:15.578526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.578488 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume" pod "afdb5260-fe44-4fda-bc18-bcc956c2d6cd" (UID: "afdb5260-fe44-4fda-bc18-bcc956c2d6cd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.580310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.580287 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv" pod "afdb5260-fe44-4fda-bc18-bcc956c2d6cd" (UID: "afdb5260-fe44-4fda-bc18-bcc956c2d6cd"). InnerVolumeSpecName "kube-api-access-dgppv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.679338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.679309 4990 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-extensions-socket-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.679338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.679332 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgppv\" (UniqueName: \"kubernetes.io/projected/afdb5260-fe44-4fda-bc18-bcc956c2d6cd-kube-api-access-dgppv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.770899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.770864 4990 generic.go:356] "Generic (PLEG): container finished" podID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" containerID="b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6" exitCode=0 Oct 30 19:15:15.771112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.770918 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:15.771112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.770960 4990 scope.go:122] "RemoveContainer" containerID="b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6" Oct 30 19:15:15.779579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.779565 4990 scope.go:122] "RemoveContainer" containerID="b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6" Oct 30 19:15:15.779854 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:15.779835 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6\": container with ID starting with b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6 not found: ID does not exist" containerID="b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6" Oct 30 19:15:15.779900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:15.779864 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6"} err="failed to get container status \"b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6\": rpc error: code = NotFound desc = could not find container \"b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6\": container with ID starting with b5d046bc08b92cacb816682fb5cb9809dd63ef29bd7d3495a4b6b5bc4bfd98f6 not found: ID does not exist" Oct 30 19:15:15.782300 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:15.782278 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:15.793954 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:15.793909 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-4745v\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-4745v" Oct 30 19:15:16.398485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:16.398453 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" path="/var/lib/kubelet/pods/afdb5260-fe44-4fda-bc18-bcc956c2d6cd/volumes" Oct 30 19:15:16.798222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:16.798087 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:16.945393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:16.945360 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.424955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:17.424923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.629727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:17.629691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:20.903074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:20.903034 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:21.087395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:21.087360 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:23.199936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:23.199890 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:23.360957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:23.360924 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:29.004328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:29.004287 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:29.148071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:29.148034 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:30.348159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.348126 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 19:15:30.348605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.348584 4990 cpu_manager.go:414] "containerMap: removing container" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" containerName="manager" Oct 30 19:15:30.348690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.348608 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" containerName="manager" Oct 30 19:15:30.348741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.348698 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdb5260-fe44-4fda-bc18-bcc956c2d6cd" containerName="manager" Oct 30 19:15:30.600851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.600782 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.605754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.605730 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 19:15:30.652750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.652727 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzmp6\"" Oct 30 19:15:30.700956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.700927 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2fg\" (UniqueName: \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.701057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.700985 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.801489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.801460 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2fg\" (UniqueName: \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.801657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.801519 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.801829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.801812 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.820103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.820067 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2fg\" (UniqueName: \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m9dxz\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:30.929888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:30.929826 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:31.269254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:31.269179 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 19:15:31.286892 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:15:31.286863 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7796eed6_f1ac_4ed3_9a48_18f44bbca748.slice/crio-1b48d8c1a320d169a336bfa5121cc56f570e7b5bbb96e5de8c380081bf63168e WatchSource:0}: Error finding container 1b48d8c1a320d169a336bfa5121cc56f570e7b5bbb96e5de8c380081bf63168e: Status 404 returned error can't find the container with id 1b48d8c1a320d169a336bfa5121cc56f570e7b5bbb96e5de8c380081bf63168e Oct 30 19:15:31.847257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:31.847224 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" event={"ID":"7796eed6-f1ac-4ed3-9a48-18f44bbca748","Type":"ContainerStarted","Data":"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26"} Oct 30 19:15:31.847676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:31.847272 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" event={"ID":"7796eed6-f1ac-4ed3-9a48-18f44bbca748","Type":"ContainerStarted","Data":"1b48d8c1a320d169a336bfa5121cc56f570e7b5bbb96e5de8c380081bf63168e"} Oct 30 19:15:31.847676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:31.847322 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:31.896344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:31.896288 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" podStartSLOduration=1.896275353 podStartE2EDuration="1.896275353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:31.895042427 +0000 UTC m=+639.606567793" watchObservedRunningTime="2026-10-30 19:15:31.896275353 +0000 UTC m=+639.607800718" Oct 30 19:15:35.094703 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:15:35.094628 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:15:42.854259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:42.854222 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 19:15:47.361185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:47.361155 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:47.527507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:47.527472 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:57.104718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:15:57.104666 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.133339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.133292 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.266157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.266117 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.404634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.404550 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.921212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.921160 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:02.924210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.924172 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:02.936807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.936781 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fg2pp\"" Oct 30 19:16:02.948370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:02.948339 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:03.043013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.042966 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67gx\" (UniqueName: \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\") pod \"authorino-57f7f8659f-8nrk4\" (UID: \"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc\") " pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:03.046390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.046358 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:03.143700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.143665 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67gx\" (UniqueName: \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\") pod \"authorino-57f7f8659f-8nrk4\" (UID: \"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc\") " pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:03.160764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.160726 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67gx\" (UniqueName: \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\") pod \"authorino-57f7f8659f-8nrk4\" (UID: \"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc\") " pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:03.184207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.184116 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:03.235059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.235021 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:03.526038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.525953 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:03.531520 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:16:03.531488 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19bdcdc2_f1d4_49ff_8fd9_e07b6d294ffc.slice/crio-cf6c412b635af889499995e7a43805db7034f5df36faf1760141bec9d9aca6b6 WatchSource:0}: Error finding container cf6c412b635af889499995e7a43805db7034f5df36faf1760141bec9d9aca6b6: Status 404 returned error can't find the container with id cf6c412b635af889499995e7a43805db7034f5df36faf1760141bec9d9aca6b6 Oct 30 19:16:03.533869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.533846 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:16:03.988838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:03.988798 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" event={"ID":"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc","Type":"ContainerStarted","Data":"cf6c412b635af889499995e7a43805db7034f5df36faf1760141bec9d9aca6b6"} Oct 30 19:16:05.100024 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:16:05.099982 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:16:06.001565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:06.001524 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" event={"ID":"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc","Type":"ContainerStarted","Data":"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40"} Oct 30 19:16:06.034926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:06.034874 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" podStartSLOduration=2.375573842 podStartE2EDuration="4.034861194s" totalImagesPullingTime="1.659287352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:06.033473319 +0000 UTC m=+673.744998685" watchObservedRunningTime="2026-10-30 19:16:06.034861194 +0000 UTC m=+673.746386559" Oct 30 19:16:06.301074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:06.301038 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:08.010906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:08.010865 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" podUID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" containerName="authorino" containerID="cri-o://af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40" gracePeriod=30 Oct 30 19:16:08.249304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:08.249278 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:08.391681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:08.391644 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\" (UniqueName: \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\") pod \"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc\" (UID: \"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc\") " Oct 30 19:16:08.393835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:08.393795 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx" pod "19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" (UID: "19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc"). InnerVolumeSpecName "kube-api-access-m67gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:08.492755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:08.492706 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m67gx\" (UniqueName: \"kubernetes.io/projected/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc-kube-api-access-m67gx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:16:09.017309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.017265 4990 generic.go:356] "Generic (PLEG): container finished" podID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" containerID="af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40" exitCode=0 Oct 30 19:16:09.017309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.017309 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" Oct 30 19:16:09.017844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.017357 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" event={"ID":"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc","Type":"ContainerDied","Data":"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40"} Oct 30 19:16:09.017844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.017398 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-8nrk4" event={"ID":"19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc","Type":"ContainerDied","Data":"cf6c412b635af889499995e7a43805db7034f5df36faf1760141bec9d9aca6b6"} Oct 30 19:16:09.017844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.017423 4990 scope.go:122] "RemoveContainer" containerID="af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40" Oct 30 19:16:09.025601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.025580 4990 scope.go:122] "RemoveContainer" containerID="af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40" Oct 30 19:16:09.025911 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:16:09.025891 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40\": container with ID starting with af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40 not found: ID does not exist" containerID="af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40" Oct 30 19:16:09.025969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.025923 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40"} err="failed to get container status \"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40\": rpc error: code = NotFound desc = could not find container \"af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40\": container with ID starting with af6129883de435adab71ccd0f408ac8ed0288d68cbc8bc91feb320515e240d40 not found: ID does not exist" Oct 30 19:16:09.060268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.060226 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:09.064192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:09.064159 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-8nrk4"] Oct 30 19:16:10.399102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:10.399050 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" path="/var/lib/kubelet/pods/19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc/volumes" Oct 30 19:16:18.979603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:18.979566 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:19.693553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:19.693516 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.507801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:21.507761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.656020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:21.655982 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:22.273040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:22.273002 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.830453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:34.830415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.971243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:34.971206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:35.094713 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:16:35.094637 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:16:35.135324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.135289 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tn26z"] Oct 30 19:16:35.135761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.135746 4990 cpu_manager.go:414] "containerMap: removing container" podUID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" containerName="authorino" Oct 30 19:16:35.135804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.135764 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" containerName="authorino" Oct 30 19:16:35.135898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.135880 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bdcdc2-f1d4-49ff-8fd9-e07b6d294ffc" containerName="authorino" Oct 30 19:16:35.138774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.138751 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:35.153242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.153215 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tn26z"] Oct 30 19:16:35.161558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.161531 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fg2pp\"" Oct 30 19:16:35.201670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.201636 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj6x\" (UniqueName: \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\") pod \"authorino-6f6944bcdd-tn26z\" (UID: \"f16a833b-af55-4a97-b348-9c2017b034e2\") " pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:35.299060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.299017 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tn26z"] Oct 30 19:16:35.299316 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:16:35.299285 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-blj6x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-tn26z" podUID="f16a833b-af55-4a97-b348-9c2017b034e2" Oct 30 19:16:35.302924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.302902 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj6x\" (UniqueName: \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\") pod \"authorino-6f6944bcdd-tn26z\" (UID: \"f16a833b-af55-4a97-b348-9c2017b034e2\") " pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:35.324010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.323980 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj6x\" (UniqueName: \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\") pod \"authorino-6f6944bcdd-tn26z\" (UID: \"f16a833b-af55-4a97-b348-9c2017b034e2\") " pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:35.327051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.327028 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hjfjg"] Oct 30 19:16:35.329483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.329462 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.343592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.343573 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:16:35.348441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.348388 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hjfjg"] Oct 30 19:16:35.404158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.404126 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.404296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.404216 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rxh\" (UniqueName: \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.472404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.472371 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hjfjg"] Oct 30 19:16:35.472600 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:16:35.472578 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-67rxh tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" podUID="36c2684f-c8fd-4f55-aac0-a6b593624320" Oct 30 19:16:35.499541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.499510 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:16:35.502063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.502042 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.505036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.505005 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.505158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.505059 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rxh\" (UniqueName: \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.507496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.507474 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.521170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.521147 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:16:35.524324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.524301 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rxh\" (UniqueName: \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\") pod \"authorino-6889cbd8c4-hjfjg\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:35.606491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.606400 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjtb7\" (UniqueName: \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.606491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.606445 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.707579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.707544 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjtb7\" (UniqueName: \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.707763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.707587 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.710188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.710156 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.724352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.724325 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjtb7\" (UniqueName: \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\") pod \"authorino-56fb8f7466-9w2jh\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:35.819414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:35.819374 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:16:36.110943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.110873 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:16:36.115527 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:16:36.115499 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaa76cef_41b3_4428_9521_76a54b17100b.slice/crio-7e1020b80e3b335fbe5b1b6204b325ba813c311349af95fa183586e495153db3 WatchSource:0}: Error finding container 7e1020b80e3b335fbe5b1b6204b325ba813c311349af95fa183586e495153db3: Status 404 returned error can't find the container with id 7e1020b80e3b335fbe5b1b6204b325ba813c311349af95fa183586e495153db3 Oct 30 19:16:36.142485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.142448 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" event={"ID":"faa76cef-41b3-4428-9521-76a54b17100b","Type":"ContainerStarted","Data":"7e1020b80e3b335fbe5b1b6204b325ba813c311349af95fa183586e495153db3"} Oct 30 19:16:36.142485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.142464 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:36.142693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.142472 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:36.149176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.149151 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:36.151638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.151615 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:36.210751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.210717 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\" (UniqueName: \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\") pod \"f16a833b-af55-4a97-b348-9c2017b034e2\" (UID: \"f16a833b-af55-4a97-b348-9c2017b034e2\") " Oct 30 19:16:36.210751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.210759 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\" (UniqueName: \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\") pod \"36c2684f-c8fd-4f55-aac0-a6b593624320\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " Oct 30 19:16:36.211030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.210864 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\" (UniqueName: \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\") pod \"36c2684f-c8fd-4f55-aac0-a6b593624320\" (UID: \"36c2684f-c8fd-4f55-aac0-a6b593624320\") " Oct 30 19:16:36.213139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.213108 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x" pod "f16a833b-af55-4a97-b348-9c2017b034e2" (UID: "f16a833b-af55-4a97-b348-9c2017b034e2"). InnerVolumeSpecName "kube-api-access-blj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:36.213139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.213122 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh" pod "36c2684f-c8fd-4f55-aac0-a6b593624320" (UID: "36c2684f-c8fd-4f55-aac0-a6b593624320"). InnerVolumeSpecName "kube-api-access-67rxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:36.213139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.213124 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert" pod "36c2684f-c8fd-4f55-aac0-a6b593624320" (UID: "36c2684f-c8fd-4f55-aac0-a6b593624320"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:36.311623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.311585 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj6x\" (UniqueName: \"kubernetes.io/projected/f16a833b-af55-4a97-b348-9c2017b034e2-kube-api-access-blj6x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:16:36.311623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.311616 4990 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36c2684f-c8fd-4f55-aac0-a6b593624320-tls-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:16:36.311623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:36.311630 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rxh\" (UniqueName: \"kubernetes.io/projected/36c2684f-c8fd-4f55-aac0-a6b593624320-kube-api-access-67rxh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:16:37.148194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.148163 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tn26z" Oct 30 19:16:37.148586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.148201 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" event={"ID":"faa76cef-41b3-4428-9521-76a54b17100b","Type":"ContainerStarted","Data":"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23"} Oct 30 19:16:37.148586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.148336 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hjfjg" Oct 30 19:16:37.184843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.184791 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" podStartSLOduration=1.586455564 podStartE2EDuration="2.184778205s" totalImagesPullingTime="598.322641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:37.183776952 +0000 UTC m=+704.895302318" watchObservedRunningTime="2026-10-30 19:16:37.184778205 +0000 UTC m=+704.896303581" Oct 30 19:16:37.255838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.255807 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tn26z"] Oct 30 19:16:37.262373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.262345 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tn26z"] Oct 30 19:16:37.312087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.312054 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hjfjg"] Oct 30 19:16:37.318274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:37.318239 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hjfjg"] Oct 30 19:16:38.398612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:38.398582 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c2684f-c8fd-4f55-aac0-a6b593624320" path="/var/lib/kubelet/pods/36c2684f-c8fd-4f55-aac0-a6b593624320/volumes" Oct 30 19:16:38.398997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:38.398809 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16a833b-af55-4a97-b348-9c2017b034e2" path="/var/lib/kubelet/pods/f16a833b-af55-4a97-b348-9c2017b034e2/volumes" Oct 30 19:16:54.466732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:54.466686 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:54.617947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:54.617912 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:54.767586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:54.767505 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:57.235102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:57.235065 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.029613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:58.029564 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.187002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:16:58.186962 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:01.506188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:01.506152 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:01.941769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:01.941730 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:05.095697 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:17:05.095619 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:17:30.202645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:30.202610 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:30.342038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:30.342008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:31.734700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:31.734668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:31.879169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:31.879140 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:32.028691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:32.028607 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:35.097123 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:17:35.095317 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:17:56.904873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:56.904822 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:17:57.348803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:17:57.348766 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:05.095034 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:18:05.094998 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:18:35.094567 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:18:35.094484 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:18:35.739870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:35.739832 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:35.962665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:35.962631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:39.416152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:39.416115 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:40.143225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:40.143192 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:53.684899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:53.684863 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:56.103361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.103329 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm"] Oct 30 19:18:56.106688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.106668 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.129033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.129010 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-48lvl\"" Oct 30 19:18:56.129033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.129015 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 19:18:56.130766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.130743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm"] Oct 30 19:18:56.162962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.162941 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:18:56.163232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.163215 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:18:56.217126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.217078 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:56.219177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219155 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfph\" (UniqueName: \"kubernetes.io/projected/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-kube-api-access-vrfph\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.219262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219191 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.219262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219212 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.219340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219297 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.219442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219424 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.219504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.219455 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320040 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320068 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320138 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfph\" (UniqueName: \"kubernetes.io/projected/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-kube-api-access-vrfph\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320161 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320176 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320201 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320522 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320550 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.320653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.320603 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.322360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.322337 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.322749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.322729 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.338180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.338151 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfph\" (UniqueName: \"kubernetes.io/projected/98ae7206-c9c4-4e25-ae72-dcb9ff5436b5-kube-api-access-vrfph\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm\" (UID: \"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.419820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.419768 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:56.712605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.712518 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm"] Oct 30 19:18:56.716079 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:18:56.716050 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ae7206_c9c4_4e25_ae72_dcb9ff5436b5.slice/crio-3953197da61f9880322f6dfe3e6213a624a64b74d9d832020584fbe63cd00b2d WatchSource:0}: Error finding container 3953197da61f9880322f6dfe3e6213a624a64b74d9d832020584fbe63cd00b2d: Status 404 returned error can't find the container with id 3953197da61f9880322f6dfe3e6213a624a64b74d9d832020584fbe63cd00b2d Oct 30 19:18:56.751212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:56.751184 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" event={"ID":"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5","Type":"ContainerStarted","Data":"3953197da61f9880322f6dfe3e6213a624a64b74d9d832020584fbe63cd00b2d"} Oct 30 19:18:57.472451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:57.472394 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.149889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:58.149812 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.764732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:58.764690 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" event={"ID":"98ae7206-c9c4-4e25-ae72-dcb9ff5436b5","Type":"ContainerStarted","Data":"691340b489adcfd2cff06ad8291bf7403fcebbb24c6c6ee4b9808d0f6cbc9010"} Oct 30 19:18:58.800936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:58.800880 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" podStartSLOduration=1.661076293 podStartE2EDuration="2.800866116s" totalImagesPullingTime="1.139789823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:58.800170648 +0000 UTC m=+846.511696016" watchObservedRunningTime="2026-10-30 19:18:58.800866116 +0000 UTC m=+846.512391481" Oct 30 19:18:59.420338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.420306 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:18:59.696914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.696800 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx"] Oct 30 19:18:59.707732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.700940 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.723898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.723868 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:18:59.732999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.732972 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx"] Oct 30 19:18:59.851466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851436 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.851868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851475 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.851868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851621 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.851868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851668 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.851868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851737 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37ddf4c-70f5-421f-864e-7305d672b1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.852007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.851870 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htlv\" (UniqueName: \"kubernetes.io/projected/b37ddf4c-70f5-421f-864e-7305d672b1a4-kube-api-access-7htlv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.866733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.866691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:59.952357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952266 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.952357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952307 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.952357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952333 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.952625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952366 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37ddf4c-70f5-421f-864e-7305d672b1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.952625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952434 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htlv\" (UniqueName: \"kubernetes.io/projected/b37ddf4c-70f5-421f-864e-7305d672b1a4-kube-api-access-7htlv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.952625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.953035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952846 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.953035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952872 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.953035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.952983 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.955262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.955230 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37ddf4c-70f5-421f-864e-7305d672b1a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.955377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.955307 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37ddf4c-70f5-421f-864e-7305d672b1a4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:18:59.974080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:18:59.974053 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htlv\" (UniqueName: \"kubernetes.io/projected/b37ddf4c-70f5-421f-864e-7305d672b1a4-kube-api-access-7htlv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p8tsx\" (UID: \"b37ddf4c-70f5-421f-864e-7305d672b1a4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:00.020938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:00.020896 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:00.321506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:00.321467 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx"] Oct 30 19:19:00.327600 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:00.327572 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37ddf4c_70f5_421f_864e_7305d672b1a4.slice/crio-45f22f2f21628a720433748cb534119234ff197d6e4b2735113e0653cf6abb54 WatchSource:0}: Error finding container 45f22f2f21628a720433748cb534119234ff197d6e4b2735113e0653cf6abb54: Status 404 returned error can't find the container with id 45f22f2f21628a720433748cb534119234ff197d6e4b2735113e0653cf6abb54 Oct 30 19:19:00.778761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:00.778729 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" event={"ID":"b37ddf4c-70f5-421f-864e-7305d672b1a4","Type":"ContainerStarted","Data":"4cee6490b38f1ed5e3dd402ac917b56f5e11bf0b3b649b55f6ad1f2057473363"} Oct 30 19:19:00.778927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:00.778786 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" event={"ID":"b37ddf4c-70f5-421f-864e-7305d672b1a4","Type":"ContainerStarted","Data":"45f22f2f21628a720433748cb534119234ff197d6e4b2735113e0653cf6abb54"} Oct 30 19:19:00.814599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:00.814552 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" podStartSLOduration=1.627586178 podStartE2EDuration="1.814534728s" totalImagesPullingTime="186.94855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:00.813186157 +0000 UTC m=+848.524711522" watchObservedRunningTime="2026-10-30 19:19:00.814534728 +0000 UTC m=+848.526060093" Oct 30 19:19:01.022059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.021993 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:01.181969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.181936 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7"] Oct 30 19:19:01.185647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.185623 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.201474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.201445 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:19:01.212516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.212488 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7"] Oct 30 19:19:01.264415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264383 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45f86\" (UniqueName: \"kubernetes.io/projected/97f8168e-4fa4-4e31-be2a-caecba1f1a46-kube-api-access-45f86\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.264579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264425 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.264632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264574 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.264632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264612 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.264725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264675 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.264765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.264726 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.302122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.302083 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:01.365586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365557 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.365586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365585 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.365769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365609 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45f86\" (UniqueName: \"kubernetes.io/projected/97f8168e-4fa4-4e31-be2a-caecba1f1a46-kube-api-access-45f86\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.365769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365630 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.365769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365672 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.365769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.365691 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.366072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.366051 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.366178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.366048 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.366178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.366129 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.368024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.367997 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f8168e-4fa4-4e31-be2a-caecba1f1a46-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.368228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.368214 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f8168e-4fa4-4e31-be2a-caecba1f1a46-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.382551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.382529 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45f86\" (UniqueName: \"kubernetes.io/projected/97f8168e-4fa4-4e31-be2a-caecba1f1a46-kube-api-access-45f86\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-ndzw7\" (UID: \"97f8168e-4fa4-4e31-be2a-caecba1f1a46\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.500232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.500203 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:01.800685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:01.800665 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7"] Oct 30 19:19:01.802860 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:01.802799 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f8168e_4fa4_4e31_be2a_caecba1f1a46.slice/crio-735c2f5090856f22924063c262ea671e2eac10aee6d242e5d96b6738a3c3ed91 WatchSource:0}: Error finding container 735c2f5090856f22924063c262ea671e2eac10aee6d242e5d96b6738a3c3ed91: Status 404 returned error can't find the container with id 735c2f5090856f22924063c262ea671e2eac10aee6d242e5d96b6738a3c3ed91 Oct 30 19:19:02.813172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:02.813136 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" event={"ID":"97f8168e-4fa4-4e31-be2a-caecba1f1a46","Type":"ContainerStarted","Data":"c35030447b1edf57a99ce5af82828178a1ce0d53e54a9e8c778d6a4f629f9ed4"} Oct 30 19:19:02.813837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:02.813803 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" event={"ID":"97f8168e-4fa4-4e31-be2a-caecba1f1a46","Type":"ContainerStarted","Data":"735c2f5090856f22924063c262ea671e2eac10aee6d242e5d96b6738a3c3ed91"} Oct 30 19:19:02.853114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:02.852597 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" podStartSLOduration=1.63975429 podStartE2EDuration="1.852579058s" totalImagesPullingTime="212.824768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:02.850918843 +0000 UTC m=+850.562444211" watchObservedRunningTime="2026-10-30 19:19:02.852579058 +0000 UTC m=+850.564104425" Oct 30 19:19:03.500342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:03.500304 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:04.608182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:04.608150 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:05.094142 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:19:05.094085 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:19:06.420350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:06.420315 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:19:06.435805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:06.435778 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:19:06.844061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:06.844013 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm" Oct 30 19:19:08.589377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.589345 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp"] Oct 30 19:19:08.593135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.593116 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.606128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.606028 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:19:08.619011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.618985 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp"] Oct 30 19:19:08.685772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.685743 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.745920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.745894 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.746040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.745927 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bfe48-6728-4136-93e2-7090d3d90481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.746040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.745953 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.746155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.746135 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnz9x\" (UniqueName: \"kubernetes.io/projected/052bfe48-6728-4136-93e2-7090d3d90481-kube-api-access-fnz9x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.746193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.746172 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.746253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.746240 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.846957 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847006 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847025 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bfe48-6728-4136-93e2-7090d3d90481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847040 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847107 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnz9x\" (UniqueName: \"kubernetes.io/projected/052bfe48-6728-4136-93e2-7090d3d90481-kube-api-access-fnz9x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847139 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847457 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847484 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.847575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.847539 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.849346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.849315 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052bfe48-6728-4136-93e2-7090d3d90481-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.849544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.849529 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052bfe48-6728-4136-93e2-7090d3d90481-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.863456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.863434 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnz9x\" (UniqueName: \"kubernetes.io/projected/052bfe48-6728-4136-93e2-7090d3d90481-kube-api-access-fnz9x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp\" (UID: \"052bfe48-6728-4136-93e2-7090d3d90481\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:08.904471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:08.904445 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:09.200956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:09.200883 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp"] Oct 30 19:19:09.204914 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:09.204890 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod052bfe48_6728_4136_93e2_7090d3d90481.slice/crio-36d8431547974295ceb6702eb8c3abfbfb6f7821c57d560ee47c9fa41bf003f7 WatchSource:0}: Error finding container 36d8431547974295ceb6702eb8c3abfbfb6f7821c57d560ee47c9fa41bf003f7: Status 404 returned error can't find the container with id 36d8431547974295ceb6702eb8c3abfbfb6f7821c57d560ee47c9fa41bf003f7 Oct 30 19:19:09.848263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:09.848233 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" event={"ID":"052bfe48-6728-4136-93e2-7090d3d90481","Type":"ContainerStarted","Data":"7d9fa01f033804f0120bd34ee7a1890494f6ec06d58a7ed9d45ad44e0618f7f6"} Oct 30 19:19:09.848644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:09.848275 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" event={"ID":"052bfe48-6728-4136-93e2-7090d3d90481","Type":"ContainerStarted","Data":"36d8431547974295ceb6702eb8c3abfbfb6f7821c57d560ee47c9fa41bf003f7"} Oct 30 19:19:09.885703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:09.885651 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podStartSLOduration=1.671148293 podStartE2EDuration="1.885637569s" totalImagesPullingTime="214.489276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:09.885270427 +0000 UTC m=+857.596795793" watchObservedRunningTime="2026-10-30 19:19:09.885637569 +0000 UTC m=+857.597162949" Oct 30 19:19:09.905051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:09.905027 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:10.022070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:10.022046 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:10.034529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:10.034506 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:10.867271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:10.867233 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p8tsx" Oct 30 19:19:11.501051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:11.501018 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:11.519414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:11.519381 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:11.869611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:11.869587 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-ndzw7" Oct 30 19:19:17.922900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:17.922842 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:19:17.926473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:17.926452 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:17.939180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:17.939157 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:19:17.951451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:17.951427 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:19:18.022374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022352 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022380 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnlh\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022409 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022485 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022539 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022574 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022593 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022637 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.022712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.022695 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.028080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.028067 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bw5jw\"" Oct 30 19:19:18.123541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123518 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123551 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123568 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnlh\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123601 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123628 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123656 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123795 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123836 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123871 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.123998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.123961 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.124186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.124114 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.124241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.124221 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.124284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.124261 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.124372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.124356 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.125964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.125945 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.126048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.125971 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.140831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.140808 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.141206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.141189 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnlh\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\") pod \"maas-default-gateway-openshift-default-6f54779779-nlrv8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.240004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.239957 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:18.561742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.561714 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:19:18.564741 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:18.564713 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2192b8c2_3fd8_4a75_9cee_bb925cef20f8.slice/crio-389ab394b0982f7040570d584209ca075747c6e4142dfbab6e9cd29aec085e6a WatchSource:0}: Error finding container 389ab394b0982f7040570d584209ca075747c6e4142dfbab6e9cd29aec085e6a: Status 404 returned error can't find the container with id 389ab394b0982f7040570d584209ca075747c6e4142dfbab6e9cd29aec085e6a Oct 30 19:19:18.894453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.894386 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" event={"ID":"2192b8c2-3fd8-4a75-9cee-bb925cef20f8","Type":"ContainerStarted","Data":"389ab394b0982f7040570d584209ca075747c6e4142dfbab6e9cd29aec085e6a"} Oct 30 19:19:18.905536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.905510 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:18.917606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:18.917582 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:19.913979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:19.913952 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:19:21.167298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:21.167271 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:21.167553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:21.167331 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:21.167553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:21.167360 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:21.913350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:21.913309 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" event={"ID":"2192b8c2-3fd8-4a75-9cee-bb925cef20f8","Type":"ContainerStarted","Data":"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9"} Oct 30 19:19:21.951776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:21.951723 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podStartSLOduration=2.351985816 podStartE2EDuration="4.951707947s" totalImagesPullingTime="2.599722131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:21.95029122 +0000 UTC m=+869.661816586" watchObservedRunningTime="2026-10-30 19:19:21.951707947 +0000 UTC m=+869.663233314" Oct 30 19:19:22.240204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:22.240126 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:22.241636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:22.241608 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:22.241759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:22.241659 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:23.241017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:23.240981 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:23.241544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:23.241040 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:24.241212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:24.241173 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:24.241688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:24.241247 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:25.240595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:25.240559 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:25.240774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:25.240610 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:26.240349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.240317 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:26.240739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.240378 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:26.433045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.433014 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56dcb668cc-fh7dx"] Oct 30 19:19:26.436470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.436453 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.453430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.453408 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56dcb668cc-fh7dx"] Oct 30 19:19:26.516747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.516670 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cda75769-7027-40c0-8288-b1dfd4ddfe8a-tls-cert\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.516747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.516705 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxf8\" (UniqueName: \"kubernetes.io/projected/cda75769-7027-40c0-8288-b1dfd4ddfe8a-kube-api-access-znxf8\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.617442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.617407 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cda75769-7027-40c0-8288-b1dfd4ddfe8a-tls-cert\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.617442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.617443 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxf8\" (UniqueName: \"kubernetes.io/projected/cda75769-7027-40c0-8288-b1dfd4ddfe8a-kube-api-access-znxf8\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.619917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.619884 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cda75769-7027-40c0-8288-b1dfd4ddfe8a-tls-cert\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.634802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.634774 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxf8\" (UniqueName: \"kubernetes.io/projected/cda75769-7027-40c0-8288-b1dfd4ddfe8a-kube-api-access-znxf8\") pod \"authorino-56dcb668cc-fh7dx\" (UID: \"cda75769-7027-40c0-8288-b1dfd4ddfe8a\") " pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:26.748346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:26.748313 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56dcb668cc-fh7dx" Oct 30 19:19:27.069481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.069443 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56dcb668cc-fh7dx"] Oct 30 19:19:27.074731 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:27.074703 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcda75769_7027_40c0_8288_b1dfd4ddfe8a.slice/crio-0b3a6b34f5157338420b85906b7ba3818d29b2b1f7ea3b6a7506d07d64644928 WatchSource:0}: Error finding container 0b3a6b34f5157338420b85906b7ba3818d29b2b1f7ea3b6a7506d07d64644928: Status 404 returned error can't find the container with id 0b3a6b34f5157338420b85906b7ba3818d29b2b1f7ea3b6a7506d07d64644928 Oct 30 19:19:27.229732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.229705 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:27.240668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.240642 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:27.241017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.240689 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:27.944189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.944144 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56dcb668cc-fh7dx" event={"ID":"cda75769-7027-40c0-8288-b1dfd4ddfe8a","Type":"ContainerStarted","Data":"6726dbb6ee4d119f737a6d71215d513b267c3690d240949b2c9ccb2e220c45c3"} Oct 30 19:19:27.944348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.944197 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56dcb668cc-fh7dx" event={"ID":"cda75769-7027-40c0-8288-b1dfd4ddfe8a","Type":"ContainerStarted","Data":"0b3a6b34f5157338420b85906b7ba3818d29b2b1f7ea3b6a7506d07d64644928"} Oct 30 19:19:27.983178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:27.983115 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56dcb668cc-fh7dx" podStartSLOduration=1.589923931 podStartE2EDuration="1.983079701s" totalImagesPullingTime="393.15577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:27.981552979 +0000 UTC m=+875.693078347" watchObservedRunningTime="2026-10-30 19:19:27.983079701 +0000 UTC m=+875.694605069" Oct 30 19:19:28.013135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.012221 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:19:28.013626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.013582 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" podUID="faa76cef-41b3-4428-9521-76a54b17100b" containerName="authorino" containerID="cri-o://4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23" gracePeriod=30 Oct 30 19:19:28.240970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.240938 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:19:28.241547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.240988 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 19:19:28.294144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.294113 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:19:28.332921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.332884 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\" (UniqueName: \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\") pod \"faa76cef-41b3-4428-9521-76a54b17100b\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " Oct 30 19:19:28.333133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.333004 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\" (UniqueName: \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\") pod \"faa76cef-41b3-4428-9521-76a54b17100b\" (UID: \"faa76cef-41b3-4428-9521-76a54b17100b\") " Oct 30 19:19:28.335680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.335651 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7" pod "faa76cef-41b3-4428-9521-76a54b17100b" (UID: "faa76cef-41b3-4428-9521-76a54b17100b"). InnerVolumeSpecName "kube-api-access-sjtb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:28.371313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.370759 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert" pod "faa76cef-41b3-4428-9521-76a54b17100b" (UID: "faa76cef-41b3-4428-9521-76a54b17100b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:28.433704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.433674 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjtb7\" (UniqueName: \"kubernetes.io/projected/faa76cef-41b3-4428-9521-76a54b17100b-kube-api-access-sjtb7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:19:28.433704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.433698 4990 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa76cef-41b3-4428-9521-76a54b17100b-tls-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:19:28.951608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.951508 4990 generic.go:356] "Generic (PLEG): container finished" podID="faa76cef-41b3-4428-9521-76a54b17100b" containerID="4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23" exitCode=0 Oct 30 19:19:28.951608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.951551 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" Oct 30 19:19:28.951831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.951594 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" event={"ID":"faa76cef-41b3-4428-9521-76a54b17100b","Type":"ContainerDied","Data":"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23"} Oct 30 19:19:28.951831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.951663 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb8f7466-9w2jh" event={"ID":"faa76cef-41b3-4428-9521-76a54b17100b","Type":"ContainerDied","Data":"7e1020b80e3b335fbe5b1b6204b325ba813c311349af95fa183586e495153db3"} Oct 30 19:19:28.951831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.951690 4990 scope.go:122] "RemoveContainer" containerID="4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23" Oct 30 19:19:28.962462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.962440 4990 scope.go:122] "RemoveContainer" containerID="4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23" Oct 30 19:19:28.962790 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:19:28.962761 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23\": container with ID starting with 4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23 not found: ID does not exist" containerID="4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23" Oct 30 19:19:28.962868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.962807 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23"} err="failed to get container status \"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23\": rpc error: code = NotFound desc = could not find container \"4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23\": container with ID starting with 4110daa11a45b9ef3d1512852a44dfe470a702dd731d7ec598abec84ba0dba23 not found: ID does not exist" Oct 30 19:19:28.981753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.981727 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:29.000343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:28.999012 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:19:29.003465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:29.003441 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56fb8f7466-9w2jh"] Oct 30 19:19:29.675315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:29.675286 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:29.676076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:29.676053 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:29.676832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:29.676812 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:19:30.398794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:30.398756 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa76cef-41b3-4428-9521-76a54b17100b" path="/var/lib/kubelet/pods/faa76cef-41b3-4428-9521-76a54b17100b/volumes" Oct 30 19:19:32.955830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.955796 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:19:32.956242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.956135 4990 cpu_manager.go:414] "containerMap: removing container" podUID="faa76cef-41b3-4428-9521-76a54b17100b" containerName="authorino" Oct 30 19:19:32.956242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.956146 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa76cef-41b3-4428-9521-76a54b17100b" containerName="authorino" Oct 30 19:19:32.956242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.956216 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa76cef-41b3-4428-9521-76a54b17100b" containerName="authorino" Oct 30 19:19:32.961526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.961505 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977606 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977644 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977675 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977856 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977920 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.977978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.977951 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.978025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.978008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.978172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.978062 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ngg\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:32.981044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:32.981020 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:19:33.079214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079187 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079225 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079247 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079276 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079327 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ngg\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079363 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079418 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079524 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079703 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079722 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079893 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.079954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.079893 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.080211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.080186 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.081704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.081683 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.082042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.082020 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.098540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.098521 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ngg\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.098724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.098705 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-g2h62\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.137731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.137708 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:33.275499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.275430 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:33.288050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.288019 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:33.598281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.598250 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:19:33.602604 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:19:33.602573 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a1adb9_d4a0_42eb_ac5e_4098b85bba68.slice/crio-76dc5fa3734cf12069d672305862902d36d99ae2147dab27520c4493515e71a7 WatchSource:0}: Error finding container 76dc5fa3734cf12069d672305862902d36d99ae2147dab27520c4493515e71a7: Status 404 returned error can't find the container with id 76dc5fa3734cf12069d672305862902d36d99ae2147dab27520c4493515e71a7 Oct 30 19:19:33.605227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.605187 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:33.605313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.605273 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:33.605356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:33.605321 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:19:34.039634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.039553 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" event={"ID":"79a1adb9-d4a0-42eb-ac5e-4098b85bba68","Type":"ContainerStarted","Data":"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027"} Oct 30 19:19:34.040021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.039651 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" event={"ID":"79a1adb9-d4a0-42eb-ac5e-4098b85bba68","Type":"ContainerStarted","Data":"76dc5fa3734cf12069d672305862902d36d99ae2147dab27520c4493515e71a7"} Oct 30 19:19:34.089736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.089674 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podStartSLOduration=2.089655524 podStartE2EDuration="2.089655524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:34.086890624 +0000 UTC m=+881.798415992" watchObservedRunningTime="2026-10-30 19:19:34.089655524 +0000 UTC m=+881.801180892" Oct 30 19:19:34.284075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.284040 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:34.284469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.284448 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:34.284639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:34.284616 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:35.123956 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:19:35.123915 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:19:35.279116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:35.277428 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:35.279116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:35.277487 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:36.275856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:36.275817 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:36.276485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:36.276457 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:37.277236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:37.276754 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:37.277236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:37.276812 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:38.286115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:38.282469 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:38.286115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:38.282528 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:39.286116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:39.283429 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:39.286116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:39.283488 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:40.276740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:40.276700 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:19:40.276914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:40.276799 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:19:41.816522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:41.816488 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:41.817043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:41.816860 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:41.817908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:41.817885 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:19:45.646432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:45.646393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:52.495555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:52.495525 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:19:52.496594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:52.496571 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:19:52.509976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:52.509954 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:19:52.511055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:52.511036 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:19:57.116903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:57.116865 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:57.586197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:57.586169 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:57.721055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:19:57.721020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:05.094402 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:20:05.094324 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:20:09.318607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:09.318577 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:09.477340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:09.477311 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:10.851997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:10.851961 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:11.023635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:11.023601 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:35.094523 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:20:35.094488 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:20:48.634888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:48.634856 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.796429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:48.796399 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:52.299698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:52.299653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:52.432488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:52.432457 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:57.698838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:20:57.698802 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:05.094688 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:21:05.094645 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:21:14.171664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.171621 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.324304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.324271 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.682134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.682087 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.691496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.691468 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:21:14.694595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.694574 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.708948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.708925 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e67b84-openshift-default-dockercfg-dvgvb\"" Oct 30 19:21:14.723692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.721743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:21:14.800349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800317 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800355 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800383 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800475 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800504 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800533 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800610 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbz4\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800725 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.800787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.800780 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.856165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.855995 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.901277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901243 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901297 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901320 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901337 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901356 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901383 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901411 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901438 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbz4\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901510 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901791 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.901908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.901884 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.902055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.902032 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.902397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.902372 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.902551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.902437 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.904073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.904053 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.904187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.904112 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.918602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.918580 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbz4\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.918964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.918943 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:14.967833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.967767 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:21:14.973116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.972874 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:14.993925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:14.993901 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:21:15.003004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.002977 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6e0c55-openshift-default-dockercfg-mndx2\"" Oct 30 19:21:15.009141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.009118 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:15.018918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.018878 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:15.103648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103606 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103666 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103733 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103776 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103821 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103847 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d894g\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103932 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.103998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.103981 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.104032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.104015 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.161025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.160996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:15.205269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205235 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205307 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205351 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205393 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205416 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d894g\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205488 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205526 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205548 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.205720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205568 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.206165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.205745 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.206519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.206488 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.206668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.206539 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.206668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.206620 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.206961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.206940 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.208657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.208595 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.208907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.208889 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.227406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.227359 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.228085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.228067 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d894g\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.290304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.290283 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:15.310957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.310927 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:15.319474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.319448 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:21:15.332060 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:15.332019 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2351615_9d67_4c7c_a594_f4038175a585.slice/crio-86c01d91debd850496731bbdd58a97da74feeca632f3d56578fa00bbb43c68d7 WatchSource:0}: Error finding container 86c01d91debd850496731bbdd58a97da74feeca632f3d56578fa00bbb43c68d7: Status 404 returned error can't find the container with id 86c01d91debd850496731bbdd58a97da74feeca632f3d56578fa00bbb43c68d7 Oct 30 19:21:15.334667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.334631 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:15.334771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.334715 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:15.334771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.334762 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:15.485363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.485278 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:15.518839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.518806 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerStarted","Data":"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72"} Oct 30 19:21:15.518999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.518867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerStarted","Data":"86c01d91debd850496731bbdd58a97da74feeca632f3d56578fa00bbb43c68d7"} Oct 30 19:21:15.557451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.557392 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podStartSLOduration=1.5573745780000001 podStartE2EDuration="1.557374578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:15.554477197 +0000 UTC m=+983.266002566" watchObservedRunningTime="2026-10-30 19:21:15.557374578 +0000 UTC m=+983.268899945" Oct 30 19:21:15.613210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.613024 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:21:15.618410 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:15.618384 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc5ee35_7072_498a_b6ca_f4b82af26f06.slice/crio-6f7c3f82f592b8ec40459fb8d84c7bbafdad83d5ce61ba2d3cccbe590a341065 WatchSource:0}: Error finding container 6f7c3f82f592b8ec40459fb8d84c7bbafdad83d5ce61ba2d3cccbe590a341065: Status 404 returned error can't find the container with id 6f7c3f82f592b8ec40459fb8d84c7bbafdad83d5ce61ba2d3cccbe590a341065 Oct 30 19:21:15.620664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.620634 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:15.620755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.620701 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:15.620755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:15.620731 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:16.010013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.009982 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:16.014966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.014941 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:16.527339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.527302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerStarted","Data":"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a"} Oct 30 19:21:16.527788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.527357 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerStarted","Data":"6f7c3f82f592b8ec40459fb8d84c7bbafdad83d5ce61ba2d3cccbe590a341065"} Oct 30 19:21:16.527788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.527571 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:16.528342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.528326 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:16.576329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:16.576288 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podStartSLOduration=2.576276249 podStartE2EDuration="2.576276249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:16.575039505 +0000 UTC m=+984.286564883" watchObservedRunningTime="2026-10-30 19:21:16.576276249 +0000 UTC m=+984.287801614" Oct 30 19:21:17.290880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:17.290841 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:17.296002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:17.295978 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:17.532491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:17.532460 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:17.533371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:17.533350 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:18.322086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:18.322043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:18.486022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:18.485979 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.196316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:19.196267 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.377397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:19.377351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.550860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:19.550810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.182229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.182198 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.248825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.248793 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 19:21:20.252152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.252135 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.267208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.267191 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9efc72\"" Oct 30 19:21:20.283595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.283576 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 19:21:20.357602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.357575 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zng6h\" (UniqueName: \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.357726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.357614 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.420969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.420924 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.459037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.458968 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zng6h\" (UniqueName: \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.459037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.459006 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.459718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.459698 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.476516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.476488 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zng6h\" (UniqueName: \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\") pod \"payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.565567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.565541 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:20.857484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.857398 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 19:21:20.861486 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:20.861450 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d56405_39e4_48bd_9623_bf65d5d279aa.slice/crio-aa9c4ad46df1b7987ec5731219b8f9804ad3f67fe89ca1521bc7b05f0cba6b75 WatchSource:0}: Error finding container aa9c4ad46df1b7987ec5731219b8f9804ad3f67fe89ca1521bc7b05f0cba6b75: Status 404 returned error can't find the container with id aa9c4ad46df1b7987ec5731219b8f9804ad3f67fe89ca1521bc7b05f0cba6b75 Oct 30 19:21:20.863465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:20.863448 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:21:21.556547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:21.556510 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:21.560153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:21.560123 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" event={"ID":"67d56405-39e4-48bd-9623-bf65d5d279aa","Type":"ContainerStarted","Data":"aa9c4ad46df1b7987ec5731219b8f9804ad3f67fe89ca1521bc7b05f0cba6b75"} Oct 30 19:21:21.704579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:21.704542 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:21.866218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:21.857404 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:21.997207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:21.997171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.055032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.055002 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:21:22.058536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.058505 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.070895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.070869 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.071039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.071009 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q986v\" (UniqueName: \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.079078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.079054 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:21:22.083344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.083320 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-6e0c55\"" Oct 30 19:21:22.128171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.128110 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:21:22.131425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.131407 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.145272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.145248 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.149028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.149010 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-6e0c55-dockercfg-4l8vf\"" Oct 30 19:21:22.151552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.151529 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:21:22.172024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.171994 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.172203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.172134 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q986v\" (UniqueName: \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.172203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.172170 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.172203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.172198 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts84d\" (UniqueName: \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.172804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.172779 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.205366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.205334 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q986v\" (UniqueName: \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\") pod \"payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.273017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.272985 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.273181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.273024 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts84d\" (UniqueName: \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.273923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.273898 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.290592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.290561 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts84d\" (UniqueName: \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\") pod \"payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.373627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.373593 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:22.459063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.446624 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:22.690684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.690651 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:21:22.692614 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:22.692572 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ca7004_923b_4f87_80de_4481a7e8f182.slice/crio-fbf584d877ae46a0bc09dc702f6f3ac453038175dc6d1820a07d0a1837339791 WatchSource:0}: Error finding container fbf584d877ae46a0bc09dc702f6f3ac453038175dc6d1820a07d0a1837339791: Status 404 returned error can't find the container with id fbf584d877ae46a0bc09dc702f6f3ac453038175dc6d1820a07d0a1837339791 Oct 30 19:21:22.760876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.760843 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:21:22.764009 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:22.763979 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047938d4_8ce7_42d8_af9a_0b909bc73524.slice/crio-0ef820468b3a880f5cb64e47ebe18f7f2ce50829bf06626b4405b0c68207929c WatchSource:0}: Error finding container 0ef820468b3a880f5cb64e47ebe18f7f2ce50829bf06626b4405b0c68207929c: Status 404 returned error can't find the container with id 0ef820468b3a880f5cb64e47ebe18f7f2ce50829bf06626b4405b0c68207929c Oct 30 19:21:22.904743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:22.904713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.028201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.028132 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 19:21:23.032135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.032110 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.047260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.047239 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5ac614-dockercfg-7f9qz\"" Oct 30 19:21:23.047382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.047285 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5ac614\"" Oct 30 19:21:23.053873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.053833 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.059313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.059293 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 19:21:23.083684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.083651 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnrn\" (UniqueName: \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.083837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.083747 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.186269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.184599 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.186269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.184692 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnrn\" (UniqueName: \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.186269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.186077 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.202511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.202355 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnrn\" (UniqueName: \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\") pod \"payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.347488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.347465 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:23.418719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.418691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.576825 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" event={"ID":"e9ca7004-923b-4f87-80de-4481a7e8f182","Type":"ContainerStarted","Data":"b6bdd85d26435592faf0b5ef8854bbc7475f84f4a96685b305b9084b162c65ef"} Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.576888 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" event={"ID":"e9ca7004-923b-4f87-80de-4481a7e8f182","Type":"ContainerStarted","Data":"fbf584d877ae46a0bc09dc702f6f3ac453038175dc6d1820a07d0a1837339791"} Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.577140 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.578781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" event={"ID":"047938d4-8ce7-42d8-af9a-0b909bc73524","Type":"ContainerStarted","Data":"16fff3d24019cf12243c1b631e89cddb58cd34ac0f63bdd8fa83644cc8519dfc"} Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.578816 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" event={"ID":"047938d4-8ce7-42d8-af9a-0b909bc73524","Type":"ContainerStarted","Data":"0ef820468b3a880f5cb64e47ebe18f7f2ce50829bf06626b4405b0c68207929c"} Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.579255 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.580864 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" event={"ID":"67d56405-39e4-48bd-9623-bf65d5d279aa","Type":"ContainerStarted","Data":"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad"} Oct 30 19:21:23.586156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.580993 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:23.613838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.613785 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" podStartSLOduration=0.963361272 podStartE2EDuration="1.613770562s" totalImagesPullingTime="650.40929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:23.612474108 +0000 UTC m=+991.323999475" watchObservedRunningTime="2026-10-30 19:21:23.613770562 +0000 UTC m=+991.325295959" Oct 30 19:21:23.647834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.647787 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" podStartSLOduration=1.342753451 podStartE2EDuration="3.64777485s" totalImagesPullingTime="2.305021399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:23.647579824 +0000 UTC m=+991.359105190" watchObservedRunningTime="2026-10-30 19:21:23.64777485 +0000 UTC m=+991.359300218" Oct 30 19:21:23.659773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.659744 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 19:21:23.664681 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:23.664656 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac42b72_87de_4b3c_b813_a0297d8f0187.slice/crio-7601a895e17fc5b7168ab8da0a7b4ee4cee992f3d26c17d6ae4c037e87936ff6 WatchSource:0}: Error finding container 7601a895e17fc5b7168ab8da0a7b4ee4cee992f3d26c17d6ae4c037e87936ff6: Status 404 returned error can't find the container with id 7601a895e17fc5b7168ab8da0a7b4ee4cee992f3d26c17d6ae4c037e87936ff6 Oct 30 19:21:23.682824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:23.682781 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" podStartSLOduration=1.139617493 podStartE2EDuration="1.682768393s" totalImagesPullingTime="543.1509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:23.681836878 +0000 UTC m=+991.393362245" watchObservedRunningTime="2026-10-30 19:21:23.682768393 +0000 UTC m=+991.394294164" Oct 30 19:21:24.572057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.572022 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:21:24.576278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.576257 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.591438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.591407 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" event={"ID":"6ac42b72-87de-4b3c-b813-a0297d8f0187","Type":"ContainerStarted","Data":"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f"} Oct 30 19:21:24.591573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.591459 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" event={"ID":"6ac42b72-87de-4b3c-b813-a0297d8f0187","Type":"ContainerStarted","Data":"7601a895e17fc5b7168ab8da0a7b4ee4cee992f3d26c17d6ae4c037e87936ff6"} Oct 30 19:21:24.591771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.591750 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:24.593206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.593008 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b97379-openshift-default-dockercfg-zcgdk\"" Oct 30 19:21:24.596525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596496 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.596818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596543 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.596818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596615 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.596818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596650 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffbg\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.596818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596687 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.597048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596861 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.597048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596907 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.597048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596968 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.597048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.596993 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.604843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.604818 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:21:24.656607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.656564 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" podStartSLOduration=1.6565482980000001 podStartE2EDuration="1.656548298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:24.654576759 +0000 UTC m=+992.366102125" watchObservedRunningTime="2026-10-30 19:21:24.656548298 +0000 UTC m=+992.368073669" Oct 30 19:21:24.684459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.684436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.697402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697378 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697418 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697439 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697455 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697479 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697496 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697535 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697569 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffbg\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697604 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697826 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697875 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.697986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.697967 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.698239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.698219 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.698469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.698452 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.699977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.699837 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.700160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.699997 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.722473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.722447 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.722593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.722577 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffbg\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:24.839834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.839746 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.891643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:24.891616 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:25.118474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.118393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:25.263671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.263599 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:21:25.267138 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:25.267107 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4289f63_a454_4f75_9741_3bf4bf835fea.slice/crio-b69233a04f7ad5f4f6685c86959376488e4674fbee5455e6ca143eb597eff0b1 WatchSource:0}: Error finding container b69233a04f7ad5f4f6685c86959376488e4674fbee5455e6ca143eb597eff0b1: Status 404 returned error can't find the container with id b69233a04f7ad5f4f6685c86959376488e4674fbee5455e6ca143eb597eff0b1 Oct 30 19:21:25.269808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.269776 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:25.269876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.269839 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:25.269876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.269868 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:25.366865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.366830 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:25.521703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.521617 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:25.601558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.601526 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerStarted","Data":"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493"} Oct 30 19:21:25.602042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.601588 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerStarted","Data":"b69233a04f7ad5f4f6685c86959376488e4674fbee5455e6ca143eb597eff0b1"} Oct 30 19:21:25.640620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.640570 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podStartSLOduration=1.640556271 podStartE2EDuration="1.640556271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:25.639780278 +0000 UTC m=+993.351305643" watchObservedRunningTime="2026-10-30 19:21:25.640556271 +0000 UTC m=+993.352081636" Oct 30 19:21:25.892798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:25.892766 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:26.898756 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:26.898727 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:27.375800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:27.375761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:27.518178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:27.518142 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:27.613966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:27.613932 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:27.615033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:27.615010 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:21:28.734438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:28.734401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.893508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:28.893478 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.072866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:29.072834 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.231777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:29.231741 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.397776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:29.397688 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.642330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:29.642293 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.894726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:29.894676 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:30.759624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.759591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:30.882565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.882534 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 19:21:30.886392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.886371 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:30.905509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.905482 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 19:21:30.917813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.917773 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:30.934825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.934801 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e67b84\"" Oct 30 19:21:30.935486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.935463 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e67b84-dockercfg-8sl6v\"" Oct 30 19:21:30.964712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.964680 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:30.964878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:30.964757 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48df\" (UniqueName: \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.065716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.065683 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.065909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.065738 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48df\" (UniqueName: \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.066457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.066434 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.082973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.082945 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48df\" (UniqueName: \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\") pod \"payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.203270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.203238 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.482913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.482836 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.516739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.516693 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 19:21:31.521237 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:31.521202 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79fb900_6199_42f1_83c2_3dcdb1671412.slice/crio-da266d0bfee9775e8a37ae7eb4919cdb4d3c0453fd4b5902f2bc26ac4e863478 WatchSource:0}: Error finding container da266d0bfee9775e8a37ae7eb4919cdb4d3c0453fd4b5902f2bc26ac4e863478: Status 404 returned error can't find the container with id da266d0bfee9775e8a37ae7eb4919cdb4d3c0453fd4b5902f2bc26ac4e863478 Oct 30 19:21:31.635847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.635814 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" event={"ID":"c79fb900-6199-42f1-83c2-3dcdb1671412","Type":"ContainerStarted","Data":"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566"} Oct 30 19:21:31.635995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.635874 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" event={"ID":"c79fb900-6199-42f1-83c2-3dcdb1671412","Type":"ContainerStarted","Data":"da266d0bfee9775e8a37ae7eb4919cdb4d3c0453fd4b5902f2bc26ac4e863478"} Oct 30 19:21:31.635995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.635925 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:31.674587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:31.674541 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" podStartSLOduration=1.674528029 podStartE2EDuration="1.674528029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:31.673754197 +0000 UTC m=+999.385279575" watchObservedRunningTime="2026-10-30 19:21:31.674528029 +0000 UTC m=+999.386053394" Oct 30 19:21:33.255960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.255921 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.402518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.402478 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.445840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.445809 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 19:21:33.449260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.449243 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.484695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.484615 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 19:21:33.488354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.488331 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b97379-dockercfg-2r4cs\"" Oct 30 19:21:33.488564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.488547 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b97379\"" Oct 30 19:21:33.500643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.500616 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxv4\" (UniqueName: \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.500983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.500957 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.602988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.602952 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxv4\" (UniqueName: \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.603198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.603058 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.603844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.603819 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.620580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.620550 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxv4\" (UniqueName: \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\") pod \"payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.693928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.689798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.762224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.762198 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:33.897610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:33.897515 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.055645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.055616 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.095321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.095126 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 19:21:34.100104 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:34.100004 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dbe448_a748_4264_ba38_8ed7dbc12e4d.slice/crio-e9f0be305b148edc181ca8cd6de4214b0520f2f825c7b8bb99455d4268f9e278 WatchSource:0}: Error finding container e9f0be305b148edc181ca8cd6de4214b0520f2f825c7b8bb99455d4268f9e278: Status 404 returned error can't find the container with id e9f0be305b148edc181ca8cd6de4214b0520f2f825c7b8bb99455d4268f9e278 Oct 30 19:21:34.425692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.425447 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.600115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.599666 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 19:21:34.600115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.599668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.601440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.601418 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:21:34.658966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.658920 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" event={"ID":"f6dbe448-a748-4264-ba38-8ed7dbc12e4d","Type":"ContainerStarted","Data":"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c"} Oct 30 19:21:34.659184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.658988 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" event={"ID":"f6dbe448-a748-4264-ba38-8ed7dbc12e4d","Type":"ContainerStarted","Data":"e9f0be305b148edc181ca8cd6de4214b0520f2f825c7b8bb99455d4268f9e278"} Oct 30 19:21:34.659656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.659633 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:34.730413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:34.730324 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" podStartSLOduration=1.7303103420000001 podStartE2EDuration="1.730310342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:34.729602182 +0000 UTC m=+1002.441127540" watchObservedRunningTime="2026-10-30 19:21:34.730310342 +0000 UTC m=+1002.441835718" Oct 30 19:21:35.123932 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:21:35.123886 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:21:35.444838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.444763 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:35.602184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.601970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:35.605077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.605043 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:21:35.605985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.605964 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 19:21:35.708626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.699849 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 19:21:35.732155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.728953 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 19:21:35.732155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.730261 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.744225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.743969 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c5449c\"" Oct 30 19:21:35.744225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.744166 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c5449c-dockercfg-g5nsp\"" Oct 30 19:21:35.833854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.833571 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.833854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.833760 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5jg\" (UniqueName: \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.935004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.934970 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.935217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.935196 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5jg\" (UniqueName: \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.936189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.936158 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.953606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.953581 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5jg\" (UniqueName: \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\") pod \"payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:35.996030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:35.995820 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:36.057060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.056799 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:36.135052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.134877 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:36.391641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.391608 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 19:21:36.396150 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:36.396123 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42371951_c7d5_4fc2_a3a0_8a8af50a948c.slice/crio-a3655f33c6e5a7ebbaa5a093ce3db29ae676c339ed362a82629ec8df2e2bd179 WatchSource:0}: Error finding container a3655f33c6e5a7ebbaa5a093ce3db29ae676c339ed362a82629ec8df2e2bd179: Status 404 returned error can't find the container with id a3655f33c6e5a7ebbaa5a093ce3db29ae676c339ed362a82629ec8df2e2bd179 Oct 30 19:21:36.439422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.439401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:36.714685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.714540 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" event={"ID":"42371951-c7d5-4fc2-a3a0-8a8af50a948c","Type":"ContainerStarted","Data":"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0"} Oct 30 19:21:36.714685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.714596 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" event={"ID":"42371951-c7d5-4fc2-a3a0-8a8af50a948c","Type":"ContainerStarted","Data":"a3655f33c6e5a7ebbaa5a093ce3db29ae676c339ed362a82629ec8df2e2bd179"} Oct 30 19:21:36.748288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:36.748239 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" podStartSLOduration=1.7482268109999999 podStartE2EDuration="1.748226811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:36.747201859 +0000 UTC m=+1004.458727225" watchObservedRunningTime="2026-10-30 19:21:36.748226811 +0000 UTC m=+1004.459752177" Oct 30 19:21:37.721347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:37.721319 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:37.852629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:37.852596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.094131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.094085 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 19:21:38.098235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.098205 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.120610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.120583 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:21:38.120797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.120774 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b97379-serving-cert\"" Oct 30 19:21:38.120929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.120894 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:21:38.120984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.120972 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:21:38.121151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.121086 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r84xv\"" Oct 30 19:21:38.129120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.123819 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 19:21:38.158535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.158503 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.158716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.158690 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589c7\" (UniqueName: \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.158781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.158742 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.158828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.158796 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.240838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.240735 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.259923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.259881 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.260076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.259965 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589c7\" (UniqueName: \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.260076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.259989 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.260076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.260005 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.260278 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:21:38.260104 4990 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-serving-cert: secret "maas-api-e2e-worker-w4-b97379-serving-cert" not found Oct 30 19:21:38.260278 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:21:38.260194 4990 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls podName:cb134f79-e7dc-42aa-9960-0e2843766d5a nodeName:}" failed. No retries permitted until 2026-10-30 19:21:38.760171305 +0000 UTC m=+1006.471696669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls") pod "maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" (UID: "cb134f79-e7dc-42aa-9960-0e2843766d5a") : secret "maas-api-e2e-worker-w4-b97379-serving-cert" not found Oct 30 19:21:38.260666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.260644 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.262499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.262478 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.279514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.279477 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589c7\" (UniqueName: \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.397075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.397037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.547045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.546959 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.764589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.764550 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:38.767735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:38.767701 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:39.011810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:39.011765 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:39.209805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:39.209751 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:39.322021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:39.321978 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 19:21:39.328315 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:39.328274 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb134f79_e7dc_42aa_9960_0e2843766d5a.slice/crio-3d1b05b906c209e1d6b7be737d7308e2902713fd78e9c6b7aae5f8871f56713a WatchSource:0}: Error finding container 3d1b05b906c209e1d6b7be737d7308e2902713fd78e9c6b7aae5f8871f56713a: Status 404 returned error can't find the container with id 3d1b05b906c209e1d6b7be737d7308e2902713fd78e9c6b7aae5f8871f56713a Oct 30 19:21:39.736722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:39.736646 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" event={"ID":"cb134f79-e7dc-42aa-9960-0e2843766d5a","Type":"ContainerStarted","Data":"3d1b05b906c209e1d6b7be737d7308e2902713fd78e9c6b7aae5f8871f56713a"} Oct 30 19:21:39.845654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:39.845615 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.005381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:40.005246 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.163122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:40.162840 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:41.959477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:41.959436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:42.102134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.102040 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:42.250254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.250218 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:42.644191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.644163 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 19:21:42.769121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.765077 4990 generic.go:356] "Generic (PLEG): container finished" podID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerID="dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a" exitCode=0 Oct 30 19:21:42.769121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.765276 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerDied","Data":"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a"} Oct 30 19:21:42.769121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.765733 4990 scope.go:122] "RemoveContainer" containerID="dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a" Oct 30 19:21:42.794484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.794374 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:42.794484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.794483 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:42.794753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.794532 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:42.797479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.797425 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" event={"ID":"cb134f79-e7dc-42aa-9960-0e2843766d5a","Type":"ContainerStarted","Data":"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94"} Oct 30 19:21:42.798150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.798059 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:42.854607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:42.854553 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" podStartSLOduration=1.5414869119999999 podStartE2EDuration="4.854538288s" totalImagesPullingTime="3.313051376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:42.85035408 +0000 UTC m=+1010.561879445" watchObservedRunningTime="2026-10-30 19:21:42.854538288 +0000 UTC m=+1010.566063657" Oct 30 19:21:43.730893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:43.730855 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:43.821275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:43.821236 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerStarted","Data":"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d"} Oct 30 19:21:44.294743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:44.294705 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:44.295154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:44.295127 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:21:44.295336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:44.295310 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:21:45.298675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:45.298630 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:21:45.299327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:45.299291 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:21:45.428306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:45.428187 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:46.291592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:46.291549 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:21:46.291924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:46.291876 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:21:46.728068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:46.728029 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 19:21:47.295123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:47.294576 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:21:47.295123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:47.294638 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:21:47.435935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:47.435881 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:47.533210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:47.533176 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:48.294967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:48.294491 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:21:48.294967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:48.294558 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:21:48.734617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:48.733953 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 19:21:49.263117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.262562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.364138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.363918 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:49.384862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.384329 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:21:49.406188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.406079 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.683190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.682971 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.834443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:49.834414 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 19:21:50.426533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:50.426488 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.596118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:50.593492 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.028262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:51.028225 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.207477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:51.207439 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:53.039520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.039429 4990 generic.go:356] "Generic (PLEG): container finished" podID="c2351615-9d67-4c7c-a594-f4038175a585" containerID="08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72" exitCode=0 Oct 30 19:21:53.039520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.039505 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerDied","Data":"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72"} Oct 30 19:21:53.039978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.039889 4990 scope.go:122] "RemoveContainer" containerID="08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72" Oct 30 19:21:53.041112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.041059 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:53.041174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.041156 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:53.041209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.041190 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:53.176714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:53.176644 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.048365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.048326 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerStarted","Data":"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c"} Oct 30 19:21:54.863418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.863384 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:21:54.870392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.870363 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.917119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.907451 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c5449c-openshift-default-dockercfg-k6524\"" Oct 30 19:21:54.922395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.922364 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:21:54.932151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.932127 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:21:54.932413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.932394 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:21:54.933268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.932769 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:54.951963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.951860 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952146 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952191 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952262 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92l2\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952289 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952317 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952356 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952533 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:54.952743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:54.952597 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.009649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.009610 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:21:55.009843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.009818 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:21:55.009920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.009858 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:21:55.053321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053285 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053327 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053416 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053464 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053491 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053521 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053560 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053612 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92l2\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053639 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053668 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053706 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.053758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053756 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053781 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053820 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053855 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053888 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053937 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053948 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.053971 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdpt\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.054428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.054057 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.054493 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.054777 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.054959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.054935 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.056369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.056346 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:55.056836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.056810 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.057217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.057197 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.104762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.104727 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92l2\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.106574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.106548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.155196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155103 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155152 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155193 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155227 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155294 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdpt\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155367 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155396 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155487 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155542 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155594 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155715 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.155931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.155907 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.156800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.156763 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.157083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.157047 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.158720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.157969 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.158720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.158325 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.191559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.191526 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.193143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.193113 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdpt\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.240026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.239981 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:55.257187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.257158 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:55.606455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.606417 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:21:55.610177 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:55.610137 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53817bca_3067_4bbc_9d96_f74a87a56c1a.slice/crio-c219ca27b1be79c4edd64ae564c74e9e42c340ee617c7558e10b0a424d5899cb WatchSource:0}: Error finding container c219ca27b1be79c4edd64ae564c74e9e42c340ee617c7558e10b0a424d5899cb: Status 404 returned error can't find the container with id c219ca27b1be79c4edd64ae564c74e9e42c340ee617c7558e10b0a424d5899cb Oct 30 19:21:55.613148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.613059 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.613284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.613196 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.613284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.613247 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.620466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.620441 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:21:55.625342 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:21:55.625316 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59350c6c_5729_4e1e_9e47_ff76ddd65c12.slice/crio-386156c756df8a42690e4d47ade9937880ac13c09935e09854c8161243581ee3 WatchSource:0}: Error finding container 386156c756df8a42690e4d47ade9937880ac13c09935e09854c8161243581ee3: Status 404 returned error can't find the container with id 386156c756df8a42690e4d47ade9937880ac13c09935e09854c8161243581ee3 Oct 30 19:21:55.628499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.628456 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.628602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.628559 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.628664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.628601 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:21:55.788113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:55.787972 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:56.010889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.010445 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:21:56.010889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.010500 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:21:56.063551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.063509 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" event={"ID":"59350c6c-5729-4e1e-9e47-ff76ddd65c12","Type":"ContainerStarted","Data":"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840"} Oct 30 19:21:56.064014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.063583 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" event={"ID":"59350c6c-5729-4e1e-9e47-ff76ddd65c12","Type":"ContainerStarted","Data":"386156c756df8a42690e4d47ade9937880ac13c09935e09854c8161243581ee3"} Oct 30 19:21:56.070519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.070486 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" event={"ID":"53817bca-3067-4bbc-9d96-f74a87a56c1a","Type":"ContainerStarted","Data":"955161f8ed4c9e0ebb8ddf1a9a2b1d0b13b5b07760efd16eae287c45f26ac8bc"} Oct 30 19:21:56.071495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.071471 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" event={"ID":"53817bca-3067-4bbc-9d96-f74a87a56c1a","Type":"ContainerStarted","Data":"c219ca27b1be79c4edd64ae564c74e9e42c340ee617c7558e10b0a424d5899cb"} Oct 30 19:21:56.106296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.106228 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podStartSLOduration=2.106206817 podStartE2EDuration="2.106206817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:56.102521685 +0000 UTC m=+1023.814047052" watchObservedRunningTime="2026-10-30 19:21:56.106206817 +0000 UTC m=+1023.817732187" Oct 30 19:21:56.141913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.141854 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podStartSLOduration=2.141839794 podStartE2EDuration="2.141839794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:56.139680522 +0000 UTC m=+1023.851205889" watchObservedRunningTime="2026-10-30 19:21:56.141839794 +0000 UTC m=+1023.853365159" Oct 30 19:21:56.241180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.241137 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:21:56.242765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.242731 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:21:56.242920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.242792 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:21:56.257941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.257903 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:21:56.258973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.258945 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:21:56.259148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.258999 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:21:56.859153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:56.856386 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:57.012513 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.012473 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:21:57.013171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.012781 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:21:57.261407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.260565 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:21:57.261407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.260634 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:21:57.261407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.261135 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:21:57.261407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.261176 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:21:57.846265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:57.846165 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:58.011562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.011523 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:21:58.012299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.011837 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:21:58.255053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.254493 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:21:58.255053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.254557 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:21:58.261133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.259359 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:21:58.261133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.259397 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:21:58.442938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.442832 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:58.757694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.756533 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:21:58.757694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.756606 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:21:58.855113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:58.839333 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:59.014638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.014537 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:21:59.014895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.014862 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 19:21:59.250126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.249646 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:21:59.250126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.249708 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:21:59.258660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.258525 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:21:59.258660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:21:59.258598 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:22:00.044785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.044745 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:22:00.045950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.045927 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:22:00.047101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.047061 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:22:00.240466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.240430 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:22:00.240715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.240690 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:22:00.259740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.259702 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:22:00.260029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.259995 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:22:00.843123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:00.843027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:01.241320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:01.241239 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:22:01.241692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:01.241319 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 19:22:01.257872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:01.257838 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:22:01.258029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:01.257886 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 19:22:02.256273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.256233 4990 generic.go:356] "Generic (PLEG): container finished" podID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerID="4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493" exitCode=0 Oct 30 19:22:02.256720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.256331 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerDied","Data":"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493"} Oct 30 19:22:02.257184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.257158 4990 scope.go:122] "RemoveContainer" containerID="4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493" Oct 30 19:22:02.258980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.258938 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:02.259127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.259046 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:02.259212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.259144 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:02.441339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.438734 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:22:02.441339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.440046 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:22:02.441339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.440705 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:22:02.487804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.487744 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:22:02.488423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.488393 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:22:02.489804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.489780 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:22:02.673165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:02.673065 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:03.193434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.193129 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:03.263683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.263649 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerStarted","Data":"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0"} Oct 30 19:22:03.816152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.816112 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:03.895117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.894487 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:22:03.895117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.894611 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:22:03.895117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.894664 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:22:03.995117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:03.994446 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.596118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:04.595049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.809527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:04.809497 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.893009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:04.892916 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:22:04.893322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:04.893278 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:22:04.990117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:04.989355 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:05.116199 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:22:05.116166 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:22:05.897348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:05.896811 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:22:05.897348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:05.896877 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:22:05.990116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:05.989932 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.406613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:06.406513 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.894042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:06.893547 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:22:06.894042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:06.893622 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:22:07.892518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:07.892480 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 19:22:07.893199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:07.893164 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 19:22:08.125650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:08.125617 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:08.565239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:08.565190 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:08.896178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:08.896072 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:22:08.896639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:08.896436 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:22:08.897139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:08.897122 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:22:09.047578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:09.047538 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:09.357874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:09.357844 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:09.851038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:09.850997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.451764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.451726 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 19:22:10.456405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.456381 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.470031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.470008 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"openshift-service-ca.crt\"" Oct 30 19:22:10.476726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.476701 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7m99\" (UniqueName: \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.476835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.476817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.476885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.476841 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.476920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.476889 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.476971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.476921 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.477058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.477035 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.480166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.480140 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"kube-root-ca.crt\"" Oct 30 19:22:10.480304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.480228 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"default-dockercfg-q2bsm\"" Oct 30 19:22:10.480366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.480315 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"simulator-w5-e67b84-kserve-self-signed-certs\"" Oct 30 19:22:10.482823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.482801 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 19:22:10.562574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.562464 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.578396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578360 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578399 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578430 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578467 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578495 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578593 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7m99\" (UniqueName: \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578839 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578907 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.578982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.578963 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.580863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.580833 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.581016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.580997 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.597795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.597769 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7m99\" (UniqueName: \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\") pod \"simulator-w5-e67b84-kserve-7966485c8b-8mtlg\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:10.770148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:10.770034 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:11.077533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.077497 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 19:22:11.080800 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:11.080771 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1bd696f_6728_43df_af50_c77f9efb9986.slice/crio-e00a8cb22d5f1d4d0e11d19f33fc098936b506f679a7bbf03101b287a8114b88 WatchSource:0}: Error finding container e00a8cb22d5f1d4d0e11d19f33fc098936b506f679a7bbf03101b287a8114b88: Status 404 returned error can't find the container with id e00a8cb22d5f1d4d0e11d19f33fc098936b506f679a7bbf03101b287a8114b88 Oct 30 19:22:11.336026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.335922 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" event={"ID":"f1bd696f-6728-43df-af50-c77f9efb9986","Type":"ContainerStarted","Data":"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0"} Oct 30 19:22:11.336026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.335977 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" event={"ID":"f1bd696f-6728-43df-af50-c77f9efb9986","Type":"ContainerStarted","Data":"e00a8cb22d5f1d4d0e11d19f33fc098936b506f679a7bbf03101b287a8114b88"} Oct 30 19:22:11.372525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.372472 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podStartSLOduration=1.3724582220000001 podStartE2EDuration="1.372458222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:11.372020992 +0000 UTC m=+1039.083546359" watchObservedRunningTime="2026-10-30 19:22:11.372458222 +0000 UTC m=+1039.083983617" Oct 30 19:22:11.617604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.617516 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:11.771063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:11.771020 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:13.258332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:13.258296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:14.884085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:14.884051 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:16.628148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:16.628114 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:17.885746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.885704 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 19:22:17.890248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.890223 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.914825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.914797 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"openshift-service-ca.crt\"" Oct 30 19:22:17.914985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.914934 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"kube-root-ca.crt\"" Oct 30 19:22:17.915613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.915591 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"simulator-w0-c5449c-kserve-self-signed-certs\"" Oct 30 19:22:17.918652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.918625 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 19:22:17.919187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.919166 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"default-dockercfg-dwctm\"" Oct 30 19:22:17.940915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.940886 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.941016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.940921 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.941016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.940957 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7hd\" (UniqueName: \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.941137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.941031 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.941137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.941071 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.941137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.941108 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:17.979144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:17.979109 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:18.042228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042193 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042233 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042267 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7hd\" (UniqueName: \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042299 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042328 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042343 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042648 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042694 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.042846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.042748 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.044675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.044650 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.044942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.044923 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.059241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.059216 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7hd\" (UniqueName: \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\") pod \"simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.206543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.206449 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:18.221187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.218436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:18.518370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:18.518293 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 19:22:18.521514 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:18.521474 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d47e18d_4bff_42b8_8cbb_3cd5b22da1d2.slice/crio-580079a5f39d5c0bdf6b7d87e84ae110b102444b67de2b3452af3c365c183fe2 WatchSource:0}: Error finding container 580079a5f39d5c0bdf6b7d87e84ae110b102444b67de2b3452af3c365c183fe2: Status 404 returned error can't find the container with id 580079a5f39d5c0bdf6b7d87e84ae110b102444b67de2b3452af3c365c183fe2 Oct 30 19:22:19.213850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.213821 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:19.383066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.383032 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" event={"ID":"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2","Type":"ContainerStarted","Data":"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd"} Oct 30 19:22:19.383282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.383077 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" event={"ID":"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2","Type":"ContainerStarted","Data":"580079a5f39d5c0bdf6b7d87e84ae110b102444b67de2b3452af3c365c183fe2"} Oct 30 19:22:19.411063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.411033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:19.420954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.420905 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podStartSLOduration=2.420892685 podStartE2EDuration="2.420892685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:19.418152207 +0000 UTC m=+1047.129677576" watchObservedRunningTime="2026-10-30 19:22:19.420892685 +0000 UTC m=+1047.132418077" Oct 30 19:22:19.629077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.629045 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:19.989789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:19.989703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:20.207379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:20.207337 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:20.770692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:20.770664 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:20.783635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:20.783610 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:21.406817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:21.406790 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:22:21.718969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:21.718855 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:23.318782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:23.318740 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:24.709294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:24.709251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:25.023648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:25.023544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.816985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:26.816946 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:27.920016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.919973 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:22:27.925146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.925123 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.943326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.943294 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:22:27.949874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.949847 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"openshift-service-ca.crt\"" Oct 30 19:22:27.950015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.949893 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"default-dockercfg-bs5pk\"" Oct 30 19:22:27.952387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.952364 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"kube-root-ca.crt\"" Oct 30 19:22:27.952490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.952436 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"premium-w6-6e0c55-kserve-self-signed-certs\"" Oct 30 19:22:27.979584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979548 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.979723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979607 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.979723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979703 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.979837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979728 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.979908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979851 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbfk\" (UniqueName: \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:27.979908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:27.979894 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.025758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.025720 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:28.080826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.080787 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.080856 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.080906 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.080930 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.081005 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbfk\" (UniqueName: \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.081037 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.081260 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.081404 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.081632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.081603 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.083899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.083857 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.084085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.084065 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.099143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.099114 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbfk\" (UniqueName: \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\") pod \"premium-w6-6e0c55-kserve-bfc4db698-jltp9\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.207684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.207594 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:28.221979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.221952 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:28.240714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.240691 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:28.366479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.365899 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:28.454435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.454408 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:22:28.551445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:28.551416 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:22:28.554742 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:28.554710 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ba216e3_c6dd_4ca2_a4b3_39401f1921d5.slice/crio-39cf9b569d72b79f60849d278ac29922271fd74523d3f6984eb9bbec3ba11675 WatchSource:0}: Error finding container 39cf9b569d72b79f60849d278ac29922271fd74523d3f6984eb9bbec3ba11675: Status 404 returned error can't find the container with id 39cf9b569d72b79f60849d278ac29922271fd74523d3f6984eb9bbec3ba11675 Oct 30 19:22:29.450552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:29.450508 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" event={"ID":"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5","Type":"ContainerStarted","Data":"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6"} Oct 30 19:22:29.451036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:29.450569 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" event={"ID":"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5","Type":"ContainerStarted","Data":"39cf9b569d72b79f60849d278ac29922271fd74523d3f6984eb9bbec3ba11675"} Oct 30 19:22:29.486255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:29.486199 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podStartSLOduration=2.486184731 podStartE2EDuration="2.486184731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:29.485153218 +0000 UTC m=+1057.196678588" watchObservedRunningTime="2026-10-30 19:22:29.486184731 +0000 UTC m=+1057.197710096" Oct 30 19:22:30.041109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:30.034837 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.241261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:30.241228 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:31.735028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:31.734988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:32.349885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:32.349848 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:33.384417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:33.384375 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:35.121848 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:22:35.121808 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:22:35.128341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:35.128306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:36.847258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:36.847222 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.241591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:38.241556 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:38.254412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:38.254384 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:38.517888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:38.517777 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.534475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:38.534444 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:22:39.834422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.834390 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 19:22:39.838628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.838605 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.862689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.862649 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 19:22:39.875138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.873075 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"kube-root-ca.crt\"" Oct 30 19:22:39.875138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.873289 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"openshift-service-ca.crt\"" Oct 30 19:22:39.875138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.873877 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"default-dockercfg-k65n6\"" Oct 30 19:22:39.875138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.873980 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"simulator-w4-b97379-kserve-self-signed-certs\"" Oct 30 19:22:39.942399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.942364 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.946558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946529 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.946715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946569 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmzv\" (UniqueName: \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.946715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946596 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.946800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946781 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.946843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946822 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:39.946922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:39.946907 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.025604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.025570 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 19:22:40.030142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.030085 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.047652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047625 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.047789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047671 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.047789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047704 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.047789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047758 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.047789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047780 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmzv\" (UniqueName: \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.048014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.047799 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.048492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.048468 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.048757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.048731 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.049772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.049747 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.051446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.051421 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.052836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.052182 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.076121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.073555 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 19:22:40.093633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.093548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmzv\" (UniqueName: \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\") pod \"simulator-w4-b97379-kserve-74794cfbf8-4qqpn\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.148538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148498 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148541 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148567 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148660 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148683 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148697 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148733 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrbs\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148758 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.148919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.148780 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.152270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.152243 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:40.249718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.249678 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrbs\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.249910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.249730 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.249910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.249842 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250055 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250113 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250170 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250179 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250248 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250288 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250314 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250447 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.250974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250948 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.251080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.250953 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.251266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.251242 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.252470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.252449 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.252625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.252599 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.266937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.266918 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrbs\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.267190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.267171 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.344721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.344594 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:40.352265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.352227 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:40.455460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.455427 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 19:22:40.457363 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:40.457337 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e95e0b_6c18_42af_a847_a25520a997ec.slice/crio-67fbf7f93ee6b1dc8ee4ced85ba75ab698c733e318b3911dffe5e4f63fbc509a WatchSource:0}: Error finding container 67fbf7f93ee6b1dc8ee4ced85ba75ab698c733e318b3911dffe5e4f63fbc509a: Status 404 returned error can't find the container with id 67fbf7f93ee6b1dc8ee4ced85ba75ab698c733e318b3911dffe5e4f63fbc509a Oct 30 19:22:40.536730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.536693 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" event={"ID":"31e95e0b-6c18-42af-a847-a25520a997ec","Type":"ContainerStarted","Data":"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea"} Oct 30 19:22:40.536888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.536744 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" event={"ID":"31e95e0b-6c18-42af-a847-a25520a997ec","Type":"ContainerStarted","Data":"67fbf7f93ee6b1dc8ee4ced85ba75ab698c733e318b3911dffe5e4f63fbc509a"} Oct 30 19:22:40.670326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.670194 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podStartSLOduration=1.670176412 podStartE2EDuration="1.670176412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:40.591992356 +0000 UTC m=+1068.303517722" watchObservedRunningTime="2026-10-30 19:22:40.670176412 +0000 UTC m=+1068.381701777" Oct 30 19:22:40.671680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.671646 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 19:22:40.675431 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:40.675405 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc174af0_337d_42ce_992f_785837dce9bb.slice/crio-716a086ba7599def27d0d1292627250b263b90a18c74738f608a2ca0d6150b69 WatchSource:0}: Error finding container 716a086ba7599def27d0d1292627250b263b90a18c74738f608a2ca0d6150b69: Status 404 returned error can't find the container with id 716a086ba7599def27d0d1292627250b263b90a18c74738f608a2ca0d6150b69 Oct 30 19:22:40.677941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.677912 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:40.678010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.677977 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:40.678010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:40.678006 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:22:41.154243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:41.154194 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:41.545494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:41.545411 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" event={"ID":"bc174af0-337d-42ce-992f-785837dce9bb","Type":"ContainerStarted","Data":"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83"} Oct 30 19:22:41.545494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:41.545457 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" event={"ID":"bc174af0-337d-42ce-992f-785837dce9bb","Type":"ContainerStarted","Data":"716a086ba7599def27d0d1292627250b263b90a18c74738f608a2ca0d6150b69"} Oct 30 19:22:42.144400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:42.144351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:42.352940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:42.352899 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:42.354201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:42.354178 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:22:42.354262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:42.354228 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:22:42.564113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:42.564054 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:43.352854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:43.352822 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:22:43.353040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:43.352888 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:22:44.050845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.050806 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.352990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.352899 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:22:44.352990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.352971 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:22:44.655731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.655631 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podStartSLOduration=4.655616541 podStartE2EDuration="4.655616541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:41.589425119 +0000 UTC m=+1069.300950486" watchObservedRunningTime="2026-10-30 19:22:44.655616541 +0000 UTC m=+1072.367141908" Oct 30 19:22:44.657879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.657853 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 19:22:44.664194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.664164 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.712898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.712869 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"premium-w5-e67b84-kserve-self-signed-certs\"" Oct 30 19:22:44.715842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.715817 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 19:22:44.717478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717455 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.717569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717496 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.717569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717532 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.717569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717558 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.717718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717623 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.717718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.717694 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggr7\" (UniqueName: \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.792661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.792621 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.818747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.818708 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.818935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.818821 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggr7\" (UniqueName: \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.818994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.818953 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.818994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.818984 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.819110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.819017 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.819110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.819042 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.819231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.819082 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.819352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.819331 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.819549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.819518 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.821453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.821428 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.821971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.821947 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:44.836425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:44.836402 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggr7\" (UniqueName: \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\") pod \"premium-w5-e67b84-kserve-947b46d59-ks87d\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:45.018358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.018259 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:45.350895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.350842 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 19:22:45.351411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.351372 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:45.353150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.353127 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:22:45.353244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.353167 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:22:45.354214 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:22:45.354187 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5222eace_2251_497e_8a42_8b506a6ac4a2.slice/crio-ea4afb0710c5e1bef635301d55f3af2af8d634f7345f6d51b9db790c17cf6a5e WatchSource:0}: Error finding container ea4afb0710c5e1bef635301d55f3af2af8d634f7345f6d51b9db790c17cf6a5e: Status 404 returned error can't find the container with id ea4afb0710c5e1bef635301d55f3af2af8d634f7345f6d51b9db790c17cf6a5e Oct 30 19:22:45.598392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.598351 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" event={"ID":"5222eace-2251-497e-8a42-8b506a6ac4a2","Type":"ContainerStarted","Data":"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c"} Oct 30 19:22:45.598392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.598397 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" event={"ID":"5222eace-2251-497e-8a42-8b506a6ac4a2","Type":"ContainerStarted","Data":"ea4afb0710c5e1bef635301d55f3af2af8d634f7345f6d51b9db790c17cf6a5e"} Oct 30 19:22:45.634629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:45.634512 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podStartSLOduration=1.634497855 podStartE2EDuration="1.634497855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:45.632247084 +0000 UTC m=+1073.343772450" watchObservedRunningTime="2026-10-30 19:22:45.634497855 +0000 UTC m=+1073.346023221" Oct 30 19:22:46.018943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:46.018845 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:46.185085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:46.185034 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:46.353524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:46.353488 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:22:46.353923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:46.353543 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:22:47.357849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:47.357817 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:47.358302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:47.358281 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:47.359189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:47.359163 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 19:22:48.023421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:48.023380 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:49.783772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:49.783721 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:50.152924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:50.152879 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:50.170300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:50.170267 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:50.643852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:50.643818 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:22:51.317219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:51.317183 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.864111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:52.864049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:53.022069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:53.022022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:54.757135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:54.757024 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.018657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:55.018570 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:55.049955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:55.049544 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:55.675883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:55.675851 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:22:56.383106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:56.383066 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:57.936578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:57.936532 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:58.076465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:58.076428 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:59.686066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.686029 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6"] Oct 30 19:22:59.692009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.691984 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.713643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.713612 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6"] Oct 30 19:22:59.715466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.715437 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:59.718828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.718803 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:22:59.719020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.718869 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:22:59.832413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.832382 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.832582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.832419 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlkj\" (UniqueName: \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.832582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.832560 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.933570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.933537 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.933570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.933574 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlkj\" (UniqueName: \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.933778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.933646 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.934351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.934327 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.935953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.935898 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:22:59.957855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:22:59.957795 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlkj\" (UniqueName: \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\") pod \"maas-api-revoke-sub-f7b3f06d-d95c6\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:23:00.015680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.015646 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:23:00.322899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.322860 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6"] Oct 30 19:23:00.330550 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:00.330521 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19090908_21bc_4663_a2f0_7e236a6f6978.slice/crio-0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb WatchSource:0}: Error finding container 0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb: Status 404 returned error can't find the container with id 0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb Oct 30 19:23:00.464680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.464640 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 19:23:00.470476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.470448 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.495052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.495019 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac614-openshift-default-dockercfg-2npc8\"" Oct 30 19:23:00.498177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.498148 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 19:23:00.642183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642076 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642146 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642238 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642275 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642394 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642433 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642547 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642609 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.642663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.642635 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfks\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.697924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.697891 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" event={"ID":"19090908-21bc-4663-a2f0-7e236a6f6978","Type":"ContainerStarted","Data":"0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb"} Oct 30 19:23:00.743453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743422 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743468 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743503 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743524 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743577 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743606 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743642 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743690 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743790 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfks\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.743900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.743835 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.744137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.744119 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.744257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.744234 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.744363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.744335 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.744478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.744463 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.746402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.746375 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.746564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.746547 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.761450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.761427 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.761544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.761512 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfks\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:00.784797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:00.784774 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:01.105554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.105517 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 19:23:01.108925 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:01.108896 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea654294_9e08_4a8c_9b6a_f33929f729a1.slice/crio-190d0960aec13550ade54a4e5350cd49c7ab510d5fa3d8cfa78902b38cc0cf23 WatchSource:0}: Error finding container 190d0960aec13550ade54a4e5350cd49c7ab510d5fa3d8cfa78902b38cc0cf23: Status 404 returned error can't find the container with id 190d0960aec13550ade54a4e5350cd49c7ab510d5fa3d8cfa78902b38cc0cf23 Oct 30 19:23:01.111892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.111854 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:01.112003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.111947 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:01.112003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.111994 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:01.456480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.456392 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:01.717167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.716747 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" event={"ID":"ea654294-9e08-4a8c-9b6a-f33929f729a1","Type":"ContainerStarted","Data":"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657"} Oct 30 19:23:01.717167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.716817 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" event={"ID":"ea654294-9e08-4a8c-9b6a-f33929f729a1","Type":"ContainerStarted","Data":"190d0960aec13550ade54a4e5350cd49c7ab510d5fa3d8cfa78902b38cc0cf23"} Oct 30 19:23:01.792273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.777551 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podStartSLOduration=1.777532109 podStartE2EDuration="1.777532109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:01.77690225 +0000 UTC m=+1089.488427617" watchObservedRunningTime="2026-10-30 19:23:01.777532109 +0000 UTC m=+1089.489057478" Oct 30 19:23:01.792273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:01.792189 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:02.787202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:02.787152 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:02.787925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:02.787806 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:03.529748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:03.529710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:03.786228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:03.785712 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:03.786228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:03.785774 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:04.786172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:04.785564 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:04.786172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:04.785633 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:05.124188 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:23:05.124137 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:23:05.796116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:05.786560 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:05.796116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:05.786625 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:05.813015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:05.812790 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" event={"ID":"19090908-21bc-4663-a2f0-7e236a6f6978","Type":"ContainerStarted","Data":"f4c0d5e01b69eaa8748358f51b9a5b755aa874eb192ca98254f3c9a17c084773"} Oct 30 19:23:05.826368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:05.826342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:05.847030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:05.846982 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" podStartSLOduration=1.861093448 podStartE2EDuration="6.846968855s" totalImagesPullingTime="4.985875407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:05.845113514 +0000 UTC m=+1093.556638881" watchObservedRunningTime="2026-10-30 19:23:05.846968855 +0000 UTC m=+1093.558494267" Oct 30 19:23:06.789172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:06.788537 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:06.789172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:06.788600 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:06.821413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:06.821377 4990 generic.go:356] "Generic (PLEG): container finished" podID="19090908-21bc-4663-a2f0-7e236a6f6978" containerID="f4c0d5e01b69eaa8748358f51b9a5b755aa874eb192ca98254f3c9a17c084773" exitCode=0 Oct 30 19:23:06.821825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:06.821451 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" event={"ID":"19090908-21bc-4663-a2f0-7e236a6f6978","Type":"ContainerDied","Data":"f4c0d5e01b69eaa8748358f51b9a5b755aa874eb192ca98254f3c9a17c084773"} Oct 30 19:23:07.613605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:07.612926 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:07.786252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:07.786199 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:23:07.786465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:07.786275 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:23:08.032997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.032966 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:23:08.132832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.132798 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\") pod \"19090908-21bc-4663-a2f0-7e236a6f6978\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " Oct 30 19:23:08.133042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.132928 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\" (UniqueName: \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\") pod \"19090908-21bc-4663-a2f0-7e236a6f6978\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " Oct 30 19:23:08.133042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.132997 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\") pod \"19090908-21bc-4663-a2f0-7e236a6f6978\" (UID: \"19090908-21bc-4663-a2f0-7e236a6f6978\") " Oct 30 19:23:08.133259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.133228 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca" pod "19090908-21bc-4663-a2f0-7e236a6f6978" (UID: "19090908-21bc-4663-a2f0-7e236a6f6978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:08.135245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.135214 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca" pod "19090908-21bc-4663-a2f0-7e236a6f6978" (UID: "19090908-21bc-4663-a2f0-7e236a6f6978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:08.135363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.135216 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj" pod "19090908-21bc-4663-a2f0-7e236a6f6978" (UID: "19090908-21bc-4663-a2f0-7e236a6f6978"). InnerVolumeSpecName "kube-api-access-zvlkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:08.235146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.235043 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:08.235146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.235079 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvlkj\" (UniqueName: \"kubernetes.io/projected/19090908-21bc-4663-a2f0-7e236a6f6978-kube-api-access-zvlkj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:08.235146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.235111 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19090908-21bc-4663-a2f0-7e236a6f6978-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:08.550048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.550012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:08.842176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.841411 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" Oct 30 19:23:08.842176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.841401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6" event={"ID":"19090908-21bc-4663-a2f0-7e236a6f6978","Type":"ContainerDied","Data":"0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb"} Oct 30 19:23:08.842176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.841533 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0712e421f384310191fe8959df65b6d376ee4e077a780f0c557a4f109014cacb" Oct 30 19:23:08.975342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.975309 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:08.975731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.975704 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:08.976951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:08.976917 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 19:23:09.192491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:09.192383 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:10.760990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:10.760954 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:12.424577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:12.424539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:13.958187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:13.955979 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:15.677729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:15.677681 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:18.012914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:18.012872 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:19.683181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:19.683141 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.556561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.556528 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 19:23:20.557285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.557258 4990 cpu_manager.go:414] "containerMap: removing container" podUID="19090908-21bc-4663-a2f0-7e236a6f6978" containerName="revoke-keys" Oct 30 19:23:20.557469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.557447 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19090908-21bc-4663-a2f0-7e236a6f6978" containerName="revoke-keys" Oct 30 19:23:20.557711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.557690 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="19090908-21bc-4663-a2f0-7e236a6f6978" containerName="revoke-keys" Oct 30 19:23:20.562960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.562943 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.579041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.579017 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 19:23:20.600861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.600834 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzg8\" (UniqueName: \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.600999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.600897 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.601052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.601034 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.601107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.601083 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.601160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.601148 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.601203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.601177 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.646790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.646754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.655468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.655447 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"premium-w4-b97379-kserve-self-signed-certs\"" Oct 30 19:23:20.702704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702667 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702771 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702789 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702810 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.702875 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzg8\" (UniqueName: \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.703261 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.703319 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.703525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.703345 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.705207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.705181 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.705323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.705308 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.720062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.720007 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzg8\" (UniqueName: \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\") pod \"premium-w4-b97379-kserve-88dd7cdc9-g4lqg\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:20.876665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:20.876568 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:21.204169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:21.202301 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 19:23:21.204230 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:21.204159 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e06bd4_e6fe_477c_8385_64ba396f02c8.slice/crio-bb9e6e451ca3dc1d9349f7e76001d43271f53d37ac031942ed82344f70f59bdf WatchSource:0}: Error finding container bb9e6e451ca3dc1d9349f7e76001d43271f53d37ac031942ed82344f70f59bdf: Status 404 returned error can't find the container with id bb9e6e451ca3dc1d9349f7e76001d43271f53d37ac031942ed82344f70f59bdf Oct 30 19:23:21.638523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:21.638489 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:21.966171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:21.966051 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" event={"ID":"a4e06bd4-e6fe-477c-8385-64ba396f02c8","Type":"ContainerStarted","Data":"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad"} Oct 30 19:23:21.966171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:21.966122 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" event={"ID":"a4e06bd4-e6fe-477c-8385-64ba396f02c8","Type":"ContainerStarted","Data":"bb9e6e451ca3dc1d9349f7e76001d43271f53d37ac031942ed82344f70f59bdf"} Oct 30 19:23:22.001437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:22.001385 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podStartSLOduration=2.001371314 podStartE2EDuration="2.001371314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:22.000332793 +0000 UTC m=+1109.711858182" watchObservedRunningTime="2026-10-30 19:23:22.001371314 +0000 UTC m=+1109.712896680" Oct 30 19:23:22.876934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:22.876903 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:23.272271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:23.272166 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:24.170663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.170624 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd"] Oct 30 19:23:24.175419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.175393 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.200362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.200327 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:23:24.200514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.200372 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:24.205102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.205060 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd"] Oct 30 19:23:24.250985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.250953 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.251162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.251110 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.251209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.251166 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljhc\" (UniqueName: \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.351830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.351785 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljhc\" (UniqueName: \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.352265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.351953 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.352265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.352116 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.352612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.352584 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.354454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.354435 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.369548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.369512 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljhc\" (UniqueName: \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\") pod \"maas-api-revoke-sub-185e145d-kk2dd\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.493389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.493307 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:24.795467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.795432 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd"] Oct 30 19:23:24.797637 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:24.797594 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698836e9_acde_4b8f_b89e_e0a768a9df0d.slice/crio-0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300 WatchSource:0}: Error finding container 0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300: Status 404 returned error can't find the container with id 0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300 Oct 30 19:23:24.962762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.962705 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:24.988420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:24.988385 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" event={"ID":"698836e9-acde-4b8f-b89e-e0a768a9df0d","Type":"ContainerStarted","Data":"0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300"} Oct 30 19:23:25.998071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:25.997976 4990 generic.go:356] "Generic (PLEG): container finished" podID="698836e9-acde-4b8f-b89e-e0a768a9df0d" containerID="2e00b3638f403453f3bb12ab710b7ffb39a40179e19912c3c21055a27fe0a4da" exitCode=0 Oct 30 19:23:25.998508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:25.998059 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" event={"ID":"698836e9-acde-4b8f-b89e-e0a768a9df0d","Type":"ContainerDied","Data":"2e00b3638f403453f3bb12ab710b7ffb39a40179e19912c3c21055a27fe0a4da"} Oct 30 19:23:26.497561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:26.497525 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:27.146035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.146007 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:27.290316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.290215 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\") pod \"698836e9-acde-4b8f-b89e-e0a768a9df0d\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " Oct 30 19:23:27.290495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.290401 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\" (UniqueName: \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\") pod \"698836e9-acde-4b8f-b89e-e0a768a9df0d\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " Oct 30 19:23:27.290495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.290455 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\") pod \"698836e9-acde-4b8f-b89e-e0a768a9df0d\" (UID: \"698836e9-acde-4b8f-b89e-e0a768a9df0d\") " Oct 30 19:23:27.290704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.290676 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca" pod "698836e9-acde-4b8f-b89e-e0a768a9df0d" (UID: "698836e9-acde-4b8f-b89e-e0a768a9df0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:27.290963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.290907 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:27.292486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.292454 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca" pod "698836e9-acde-4b8f-b89e-e0a768a9df0d" (UID: "698836e9-acde-4b8f-b89e-e0a768a9df0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:27.292639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.292615 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc" pod "698836e9-acde-4b8f-b89e-e0a768a9df0d" (UID: "698836e9-acde-4b8f-b89e-e0a768a9df0d"). InnerVolumeSpecName "kube-api-access-lljhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:27.392145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.392076 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljhc\" (UniqueName: \"kubernetes.io/projected/698836e9-acde-4b8f-b89e-e0a768a9df0d-kube-api-access-lljhc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:27.392145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:27.392138 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698836e9-acde-4b8f-b89e-e0a768a9df0d-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:28.026634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:28.026596 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" event={"ID":"698836e9-acde-4b8f-b89e-e0a768a9df0d","Type":"ContainerDied","Data":"0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300"} Oct 30 19:23:28.027018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:28.027001 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf0dd715c5d7a9c3dd75120f86e2ee5cf2ef9d3b7d8b69248a5f254e4ef1300" Oct 30 19:23:28.027302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:28.027287 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd" Oct 30 19:23:28.134142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:28.131643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:30.001414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.001377 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:23:30.001921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.001905 4990 cpu_manager.go:414] "containerMap: removing container" podUID="698836e9-acde-4b8f-b89e-e0a768a9df0d" containerName="revoke-keys" Oct 30 19:23:30.001974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.001923 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698836e9-acde-4b8f-b89e-e0a768a9df0d" containerName="revoke-keys" Oct 30 19:23:30.002009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.002000 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="698836e9-acde-4b8f-b89e-e0a768a9df0d" containerName="revoke-keys" Oct 30 19:23:30.005596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.005564 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.020347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.020324 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9efc72-openshift-default-dockercfg-jrkpb\"" Oct 30 19:23:30.034189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.034165 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:23:30.158296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158257 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158340 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158411 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158443 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158616 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158654 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158772 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctv2\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.158908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.158873 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260611 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctv2\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260656 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260747 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260789 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260817 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260875 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.260951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260899 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.260979 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261008 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261471 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261574 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261596 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261578 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.261759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.261664 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.263349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.263323 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.263822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.263799 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.274607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.274580 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:23:30.279597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.279574 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.280773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.280751 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.281618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.281597 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctv2\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.291728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.291701 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-249dd4-openshift-default-dockercfg-gbt9k\"" Oct 30 19:23:30.302178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.302080 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:23:30.320121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.319172 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:30.463689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.463652 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.463861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.463703 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.463861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.463812 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.463982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.463863 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.464038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.463992 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.464150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.464132 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjc7\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.464216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.464184 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.464278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.464255 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.464313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.464295 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.565242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.565211 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.565480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.565281 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.565480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.565319 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.565480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.565403 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjc7\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566050 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566140 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566158 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566439 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566168 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566517 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566464 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566569 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.566869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.566833 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.567133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.567113 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.568580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.568559 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.569137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.569106 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.578430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.573729 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:30.588378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.587387 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjc7\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.588378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.587597 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.608066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.608037 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:30.652185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.652160 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:23:30.655158 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:30.655127 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cc01d8_2100_4da2_8908_b792cc280149.slice/crio-7841bfd8cb8ab7758a8e42e2ac661eb43d954e09f49f27d082a301ee85ac8871 WatchSource:0}: Error finding container 7841bfd8cb8ab7758a8e42e2ac661eb43d954e09f49f27d082a301ee85ac8871: Status 404 returned error can't find the container with id 7841bfd8cb8ab7758a8e42e2ac661eb43d954e09f49f27d082a301ee85ac8871 Oct 30 19:23:30.657676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.657644 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:30.657753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.657711 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:30.657753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.657742 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:30.876842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.876758 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:30.890251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.890216 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:30.916993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.916964 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:23:30.919711 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:30.919686 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f764c77_0f46_4a4d_ba19_8f7f3cbbc7d3.slice/crio-c082df2530a005c8b47baf360111c03b84129387a5c211cc3b59cfe947ed110f WatchSource:0}: Error finding container c082df2530a005c8b47baf360111c03b84129387a5c211cc3b59cfe947ed110f: Status 404 returned error can't find the container with id c082df2530a005c8b47baf360111c03b84129387a5c211cc3b59cfe947ed110f Oct 30 19:23:30.922356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.922323 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:30.922430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.922393 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:30.922468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:30.922428 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:23:31.070420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.070377 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" event={"ID":"d5cc01d8-2100-4da2-8908-b792cc280149","Type":"ContainerStarted","Data":"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2"} Oct 30 19:23:31.070905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.070472 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" event={"ID":"d5cc01d8-2100-4da2-8908-b792cc280149","Type":"ContainerStarted","Data":"7841bfd8cb8ab7758a8e42e2ac661eb43d954e09f49f27d082a301ee85ac8871"} Oct 30 19:23:31.074058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.073815 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" event={"ID":"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3","Type":"ContainerStarted","Data":"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79"} Oct 30 19:23:31.074058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.073858 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" event={"ID":"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3","Type":"ContainerStarted","Data":"c082df2530a005c8b47baf360111c03b84129387a5c211cc3b59cfe947ed110f"} Oct 30 19:23:31.087197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.087149 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:23:31.112771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.112698 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podStartSLOduration=2.112678953 podStartE2EDuration="2.112678953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:31.107540536 +0000 UTC m=+1118.819065906" watchObservedRunningTime="2026-10-30 19:23:31.112678953 +0000 UTC m=+1118.824204321" Oct 30 19:23:31.148600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.148504 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podStartSLOduration=1.148491344 podStartE2EDuration="1.148491344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:31.145121408 +0000 UTC m=+1118.856646772" watchObservedRunningTime="2026-10-30 19:23:31.148491344 +0000 UTC m=+1118.860016710" Oct 30 19:23:31.321375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.321342 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:31.321698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.321676 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:31.321855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.321833 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:31.613191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.613148 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:31.613484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.613307 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:31.613639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:31.613618 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:32.321586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:32.321544 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:32.322219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:32.322183 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:32.430366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:32.426613 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:32.610173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:32.608463 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:32.610173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:32.608528 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:33.321590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:33.321547 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:33.322145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:33.321620 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:33.609181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:33.608556 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:33.609181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:33.608613 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:34.132511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.132059 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:34.322568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.322527 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:34.323220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.323165 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:34.622445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.622389 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:34.622646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.622470 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:34.756247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.756211 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx"] Oct 30 19:23:34.760288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.760267 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.788561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.788523 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx"] Oct 30 19:23:34.789427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.789398 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:23:34.789590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.789427 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.789698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.789684 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:34.790040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.789927 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.790230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.790208 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqb6\" (UniqueName: \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.911031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.910930 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.911655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.911633 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqb6\" (UniqueName: \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.912412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.912382 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.926875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.926838 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.935101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.935061 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqb6\" (UniqueName: \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:34.935511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:34.935488 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b4937e-5lzqx\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:35.139135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.138768 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:35.154635 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:23:35.154610 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:23:35.347626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.328487 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:35.347626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.328545 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:35.627160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.626820 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:35.627160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.626878 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:35.628521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.628040 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx"] Oct 30 19:23:35.635216 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:35.635174 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf524915f_1d34_4521_a6be_b488178c1b78.slice/crio-ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb WatchSource:0}: Error finding container ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb: Status 404 returned error can't find the container with id ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb Oct 30 19:23:35.830304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:35.830269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:36.148341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.148302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" event={"ID":"f524915f-1d34-4521-a6be-b488178c1b78","Type":"ContainerStarted","Data":"49d66f0ca5e110ca8ae6f2d407183cda053f6701aca5dcda891e573abcde7028"} Oct 30 19:23:36.148527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.148372 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" event={"ID":"f524915f-1d34-4521-a6be-b488178c1b78","Type":"ContainerStarted","Data":"ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb"} Oct 30 19:23:36.185170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.184552 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" podStartSLOduration=2.1845316869999998 podStartE2EDuration="2.184531687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:36.183472726 +0000 UTC m=+1123.894998093" watchObservedRunningTime="2026-10-30 19:23:36.184531687 +0000 UTC m=+1123.896057053" Oct 30 19:23:36.321259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.321216 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:36.321535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.321509 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:36.609120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.608530 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:36.609120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:36.608595 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:37.159150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.159037 4990 generic.go:356] "Generic (PLEG): container finished" podID="f524915f-1d34-4521-a6be-b488178c1b78" containerID="49d66f0ca5e110ca8ae6f2d407183cda053f6701aca5dcda891e573abcde7028" exitCode=0 Oct 30 19:23:37.159150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.159119 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" event={"ID":"f524915f-1d34-4521-a6be-b488178c1b78","Type":"ContainerDied","Data":"49d66f0ca5e110ca8ae6f2d407183cda053f6701aca5dcda891e573abcde7028"} Oct 30 19:23:37.320512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.320474 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:23:37.320721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.320538 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 19:23:37.435789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.435697 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:37.609408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.609366 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:23:37.610029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:37.609992 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 19:23:38.316523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.316498 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:38.324311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.324289 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:38.324576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.324557 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:38.325544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.325524 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:23:38.429278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.429203 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\") pod \"f524915f-1d34-4521-a6be-b488178c1b78\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " Oct 30 19:23:38.429432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.429290 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\") pod \"f524915f-1d34-4521-a6be-b488178c1b78\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " Oct 30 19:23:38.429432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.429369 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\" (UniqueName: \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\") pod \"f524915f-1d34-4521-a6be-b488178c1b78\" (UID: \"f524915f-1d34-4521-a6be-b488178c1b78\") " Oct 30 19:23:38.429676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.429644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca" pod "f524915f-1d34-4521-a6be-b488178c1b78" (UID: "f524915f-1d34-4521-a6be-b488178c1b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:38.429887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.429865 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:38.431601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.431564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca" pod "f524915f-1d34-4521-a6be-b488178c1b78" (UID: "f524915f-1d34-4521-a6be-b488178c1b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:38.431712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.431694 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6" pod "f524915f-1d34-4521-a6be-b488178c1b78" (UID: "f524915f-1d34-4521-a6be-b488178c1b78"). InnerVolumeSpecName "kube-api-access-ctqb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:38.530794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.530748 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqb6\" (UniqueName: \"kubernetes.io/projected/f524915f-1d34-4521-a6be-b488178c1b78-kube-api-access-ctqb6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:38.530794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.530785 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f524915f-1d34-4521-a6be-b488178c1b78-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:38.612781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.612745 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:38.613296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.613276 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:38.613954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:38.613934 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:23:39.101766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:39.101671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:39.172103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:39.172059 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" Oct 30 19:23:39.172277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:39.172047 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx" event={"ID":"f524915f-1d34-4521-a6be-b488178c1b78","Type":"ContainerDied","Data":"ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb"} Oct 30 19:23:39.172277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:39.172157 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1ed230e9de10f96abea20464cf677ce104f569b807353ebf2bbd5c9048b4fb" Oct 30 19:23:40.650589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:40.650539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:42.263149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:42.260049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:44.571700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:44.571660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:44.786191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:44.786147 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:44.934257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:44.934163 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:45.072893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.072855 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6"] Oct 30 19:23:45.073650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.073622 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f524915f-1d34-4521-a6be-b488178c1b78" containerName="revoke-keys" Oct 30 19:23:45.073650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.073647 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f524915f-1d34-4521-a6be-b488178c1b78" containerName="revoke-keys" Oct 30 19:23:45.073882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.073807 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f524915f-1d34-4521-a6be-b488178c1b78" containerName="revoke-keys" Oct 30 19:23:45.078116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.078077 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.095597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.095567 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6"] Oct 30 19:23:45.100527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.100501 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:45.100681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.100639 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:23:45.176879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.176844 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wrh\" (UniqueName: \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.177056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.176996 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.177158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.177133 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.278810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.278723 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.278976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.278876 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.279044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.279004 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wrh\" (UniqueName: \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.279539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.279511 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.282005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.281979 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.296194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.296166 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wrh\" (UniqueName: \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\") pod \"maas-api-revoke-sub-c219201c-8cqs6\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.392812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.392769 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:45.698004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:45.697963 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6"] Oct 30 19:23:45.701395 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:45.701354 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d06973b_436d_4b9e_a980_af02ca2f1204.slice/crio-4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae WatchSource:0}: Error finding container 4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae: Status 404 returned error can't find the container with id 4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae Oct 30 19:23:46.269649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:46.269544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" event={"ID":"9d06973b-436d-4b9e-a980-af02ca2f1204","Type":"ContainerStarted","Data":"eec48e5767547db0bbdc9bd66fed6549a75061da84cb0bcfddf6cd2942ed36f3"} Oct 30 19:23:46.269649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:46.269614 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" event={"ID":"9d06973b-436d-4b9e-a980-af02ca2f1204","Type":"ContainerStarted","Data":"4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae"} Oct 30 19:23:46.276346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:46.272762 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:46.303469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:46.303421 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" podStartSLOduration=1.303407175 podStartE2EDuration="1.303407175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:46.30331906 +0000 UTC m=+1134.014844426" watchObservedRunningTime="2026-10-30 19:23:46.303407175 +0000 UTC m=+1134.014932540" Oct 30 19:23:47.279294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:47.279255 4990 generic.go:356] "Generic (PLEG): container finished" podID="9d06973b-436d-4b9e-a980-af02ca2f1204" containerID="eec48e5767547db0bbdc9bd66fed6549a75061da84cb0bcfddf6cd2942ed36f3" exitCode=0 Oct 30 19:23:47.279716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:47.279351 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" event={"ID":"9d06973b-436d-4b9e-a980-af02ca2f1204","Type":"ContainerDied","Data":"eec48e5767547db0bbdc9bd66fed6549a75061da84cb0bcfddf6cd2942ed36f3"} Oct 30 19:23:48.129900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.129858 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:48.449065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.449041 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:48.534223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.534192 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\" (UniqueName: \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\") pod \"9d06973b-436d-4b9e-a980-af02ca2f1204\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " Oct 30 19:23:48.534392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.534247 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\") pod \"9d06973b-436d-4b9e-a980-af02ca2f1204\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " Oct 30 19:23:48.534392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.534379 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\") pod \"9d06973b-436d-4b9e-a980-af02ca2f1204\" (UID: \"9d06973b-436d-4b9e-a980-af02ca2f1204\") " Oct 30 19:23:48.534644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.534611 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca" pod "9d06973b-436d-4b9e-a980-af02ca2f1204" (UID: "9d06973b-436d-4b9e-a980-af02ca2f1204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:48.534875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.534855 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:48.536433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.536406 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh" pod "9d06973b-436d-4b9e-a980-af02ca2f1204" (UID: "9d06973b-436d-4b9e-a980-af02ca2f1204"). InnerVolumeSpecName "kube-api-access-w2wrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:48.536433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.536420 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca" pod "9d06973b-436d-4b9e-a980-af02ca2f1204" (UID: "9d06973b-436d-4b9e-a980-af02ca2f1204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:48.635689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.635654 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d06973b-436d-4b9e-a980-af02ca2f1204-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:48.635689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.635691 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wrh\" (UniqueName: \"kubernetes.io/projected/9d06973b-436d-4b9e-a980-af02ca2f1204-kube-api-access-w2wrh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:48.999875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:48.999841 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq"] Oct 30 19:23:49.000394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.000380 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9d06973b-436d-4b9e-a980-af02ca2f1204" containerName="revoke-keys" Oct 30 19:23:49.000437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.000395 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d06973b-436d-4b9e-a980-af02ca2f1204" containerName="revoke-keys" Oct 30 19:23:49.000495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.000486 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d06973b-436d-4b9e-a980-af02ca2f1204" containerName="revoke-keys" Oct 30 19:23:49.003736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.003720 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.022690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.022642 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq"] Oct 30 19:23:49.142238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.142204 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.142238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.142242 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.142443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.142259 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgwt\" (UniqueName: \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.243880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.243831 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.244135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.243889 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.244135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.243926 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgwt\" (UniqueName: \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.244658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.244637 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.246499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.246470 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.265537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.265472 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgwt\" (UniqueName: \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\") pod \"maas-api-revoke-sub-75479acf-dlndq\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.294496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.294459 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" event={"ID":"9d06973b-436d-4b9e-a980-af02ca2f1204","Type":"ContainerDied","Data":"4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae"} Oct 30 19:23:49.294641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.294525 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb36e26133b0d11a3853e28b09b4638cdedbafd53091e986e477ed579cf00ae" Oct 30 19:23:49.294641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.294478 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6" Oct 30 19:23:49.318139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.318115 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:49.617945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.617907 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq"] Oct 30 19:23:49.619163 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:23:49.619136 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8af8c92_6761_409c_85fd_1f6f4fd151f2.slice/crio-4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296 WatchSource:0}: Error finding container 4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296: Status 404 returned error can't find the container with id 4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296 Oct 30 19:23:49.815430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:49.815224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:50.302245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:50.302212 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" event={"ID":"a8af8c92-6761-409c-85fd-1f6f4fd151f2","Type":"ContainerStarted","Data":"9016240cb06f219852e83e17215a33b265357d92eb241aeae5063d25a4169dc7"} Oct 30 19:23:50.302412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:50.302263 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" event={"ID":"a8af8c92-6761-409c-85fd-1f6f4fd151f2","Type":"ContainerStarted","Data":"4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296"} Oct 30 19:23:50.342219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:50.342157 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" podStartSLOduration=2.3421377 podStartE2EDuration="2.3421377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:50.336956997 +0000 UTC m=+1138.048482363" watchObservedRunningTime="2026-10-30 19:23:50.3421377 +0000 UTC m=+1138.053663067" Oct 30 19:23:51.310510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:51.310469 4990 generic.go:356] "Generic (PLEG): container finished" podID="a8af8c92-6761-409c-85fd-1f6f4fd151f2" containerID="9016240cb06f219852e83e17215a33b265357d92eb241aeae5063d25a4169dc7" exitCode=0 Oct 30 19:23:51.311029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:51.310558 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" event={"ID":"a8af8c92-6761-409c-85fd-1f6f4fd151f2","Type":"ContainerDied","Data":"9016240cb06f219852e83e17215a33b265357d92eb241aeae5063d25a4169dc7"} Oct 30 19:23:51.441364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:51.441296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:52.473851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.473820 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:52.586188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.586069 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\") pod \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " Oct 30 19:23:52.586329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.586194 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\" (UniqueName: \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\") pod \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " Oct 30 19:23:52.586329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.586320 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\") pod \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\" (UID: \"a8af8c92-6761-409c-85fd-1f6f4fd151f2\") " Oct 30 19:23:52.586723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.586694 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca" pod "a8af8c92-6761-409c-85fd-1f6f4fd151f2" (UID: "a8af8c92-6761-409c-85fd-1f6f4fd151f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:52.586890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.586865 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:52.588356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.588328 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt" pod "a8af8c92-6761-409c-85fd-1f6f4fd151f2" (UID: "a8af8c92-6761-409c-85fd-1f6f4fd151f2"). InnerVolumeSpecName "kube-api-access-8jgwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:52.588472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.588384 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca" pod "a8af8c92-6761-409c-85fd-1f6f4fd151f2" (UID: "a8af8c92-6761-409c-85fd-1f6f4fd151f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:52.688254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.688220 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af8c92-6761-409c-85fd-1f6f4fd151f2-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:52.688254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:52.688251 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgwt\" (UniqueName: \"kubernetes.io/projected/a8af8c92-6761-409c-85fd-1f6f4fd151f2-kube-api-access-8jgwt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:23:53.128057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:53.128008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:53.326511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:53.326475 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" Oct 30 19:23:53.326684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:53.326511 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq" event={"ID":"a8af8c92-6761-409c-85fd-1f6f4fd151f2","Type":"ContainerDied","Data":"4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296"} Oct 30 19:23:53.326684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:53.326562 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd6c5b6e5c09a4bb7aafea613a757860aff51394ba0879f21d9a777e44e0296" Oct 30 19:23:54.750243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:54.750188 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:56.590116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:56.589887 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.045054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:58.045020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.206461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:58.206426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:59.378682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.378647 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm"] Oct 30 19:23:59.379176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.379128 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a8af8c92-6761-409c-85fd-1f6f4fd151f2" containerName="revoke-keys" Oct 30 19:23:59.379176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.379140 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8af8c92-6761-409c-85fd-1f6f4fd151f2" containerName="revoke-keys" Oct 30 19:23:59.379301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.379242 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8af8c92-6761-409c-85fd-1f6f4fd151f2" containerName="revoke-keys" Oct 30 19:23:59.383788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.383769 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.397844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.397826 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:59.398175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.398152 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:23:59.406227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.406192 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm"] Oct 30 19:23:59.476506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.476467 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2pn\" (UniqueName: \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.476665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.476647 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.476723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.476708 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.577458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.577416 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.577626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.577491 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.577626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.577589 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2pn\" (UniqueName: \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.578152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.578127 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.580080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.580049 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.595346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.595319 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2pn\" (UniqueName: \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\") pod \"maas-api-revoke-sub-c5bb6ff7-z74wm\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.697386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.697305 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:23:59.989215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:23:59.989022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:00.005940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:00.005907 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm"] Oct 30 19:24:00.007429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:00.007399 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380c5dcf_74cd_426e_a640_a3252bc7b694.slice/crio-11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3 WatchSource:0}: Error finding container 11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3: Status 404 returned error can't find the container with id 11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3 Oct 30 19:24:00.378326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:00.378288 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" event={"ID":"380c5dcf-74cd-426e-a640-a3252bc7b694","Type":"ContainerStarted","Data":"24111ebde3482af841a2c000e47a34a2708d5b4f52b474ec63d84d79479aed03"} Oct 30 19:24:00.378529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:00.378356 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" event={"ID":"380c5dcf-74cd-426e-a640-a3252bc7b694","Type":"ContainerStarted","Data":"11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3"} Oct 30 19:24:00.414382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:00.414318 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" podStartSLOduration=1.414302272 podStartE2EDuration="1.414302272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:00.412578258 +0000 UTC m=+1148.124103625" watchObservedRunningTime="2026-10-30 19:24:00.414302272 +0000 UTC m=+1148.125827644" Oct 30 19:24:01.389691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:01.389656 4990 generic.go:356] "Generic (PLEG): container finished" podID="380c5dcf-74cd-426e-a640-a3252bc7b694" containerID="24111ebde3482af841a2c000e47a34a2708d5b4f52b474ec63d84d79479aed03" exitCode=0 Oct 30 19:24:01.389876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:01.389731 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" event={"ID":"380c5dcf-74cd-426e-a640-a3252bc7b694","Type":"ContainerDied","Data":"24111ebde3482af841a2c000e47a34a2708d5b4f52b474ec63d84d79479aed03"} Oct 30 19:24:01.697809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:01.697699 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:02.550645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.550614 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:24:02.618206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.618168 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\") pod \"380c5dcf-74cd-426e-a640-a3252bc7b694\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " Oct 30 19:24:02.618397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.618220 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\") pod \"380c5dcf-74cd-426e-a640-a3252bc7b694\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " Oct 30 19:24:02.618397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.618299 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\" (UniqueName: \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\") pod \"380c5dcf-74cd-426e-a640-a3252bc7b694\" (UID: \"380c5dcf-74cd-426e-a640-a3252bc7b694\") " Oct 30 19:24:02.618606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.618578 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca" pod "380c5dcf-74cd-426e-a640-a3252bc7b694" (UID: "380c5dcf-74cd-426e-a640-a3252bc7b694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:02.618715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.618698 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:02.620535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.620501 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca" pod "380c5dcf-74cd-426e-a640-a3252bc7b694" (UID: "380c5dcf-74cd-426e-a640-a3252bc7b694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:02.620535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.620512 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn" pod "380c5dcf-74cd-426e-a640-a3252bc7b694" (UID: "380c5dcf-74cd-426e-a640-a3252bc7b694"). InnerVolumeSpecName "kube-api-access-4w2pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:02.719937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.719842 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w2pn\" (UniqueName: \"kubernetes.io/projected/380c5dcf-74cd-426e-a640-a3252bc7b694-kube-api-access-4w2pn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:02.719937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:02.719879 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380c5dcf-74cd-426e-a640-a3252bc7b694-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:03.326697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:03.326659 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:03.406697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:03.406666 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" Oct 30 19:24:03.406697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:03.406684 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm" event={"ID":"380c5dcf-74cd-426e-a640-a3252bc7b694","Type":"ContainerDied","Data":"11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3"} Oct 30 19:24:03.406955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:03.406736 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11189a3e9174dc92e9fa4e023831e94238d7d48714f84fe6fcf40b684a2675c3" Oct 30 19:24:05.260115 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:24:05.259327 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:24:05.349116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:05.341995 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:07.158659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:07.158623 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.919489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:08.919452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:10.355017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.353547 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:10.508809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.508762 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:10.604345 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.604310 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj"] Oct 30 19:24:10.604818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.604805 4990 cpu_manager.go:414] "containerMap: removing container" podUID="380c5dcf-74cd-426e-a640-a3252bc7b694" containerName="revoke-keys" Oct 30 19:24:10.604862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.604819 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380c5dcf-74cd-426e-a640-a3252bc7b694" containerName="revoke-keys" Oct 30 19:24:10.604918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.604909 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="380c5dcf-74cd-426e-a640-a3252bc7b694" containerName="revoke-keys" Oct 30 19:24:10.608141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.608101 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.626409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.626374 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj"] Oct 30 19:24:10.632635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.632613 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:24:10.633006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.632985 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:24:10.650524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.650495 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.650660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.650616 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.650704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.650672 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zd7w\" (UniqueName: \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.752011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.751969 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.752228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.752085 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.752228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.752161 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zd7w\" (UniqueName: \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.752683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.752661 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.754310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.754286 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.768858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.768833 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zd7w\" (UniqueName: \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\") pod \"maas-api-revoke-sub-57eba0bf-fnwmj\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:10.921461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:10.921384 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:11.224881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:11.224848 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj"] Oct 30 19:24:11.227525 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:11.227496 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648c22d_6c5b_47f4_b7b1_43d52052af2d.slice/crio-c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3 WatchSource:0}: Error finding container c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3: Status 404 returned error can't find the container with id c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3 Oct 30 19:24:11.478342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:11.478258 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" event={"ID":"d648c22d-6c5b-47f4-b7b1-43d52052af2d","Type":"ContainerStarted","Data":"c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3"} Oct 30 19:24:12.137199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:12.137160 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:12.491654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:12.491560 4990 generic.go:356] "Generic (PLEG): container finished" podID="d648c22d-6c5b-47f4-b7b1-43d52052af2d" containerID="843c653642bab2597a4768395ed222471215d0a9366ef039c66a95941b5e6491" exitCode=0 Oct 30 19:24:12.492332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:12.492305 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" event={"ID":"d648c22d-6c5b-47f4-b7b1-43d52052af2d","Type":"ContainerDied","Data":"843c653642bab2597a4768395ed222471215d0a9366ef039c66a95941b5e6491"} Oct 30 19:24:13.655828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.655803 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:13.790529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.790442 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\") pod \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " Oct 30 19:24:13.790529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.790491 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\") pod \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " Oct 30 19:24:13.790737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.790713 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\" (UniqueName: \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\") pod \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\" (UID: \"d648c22d-6c5b-47f4-b7b1-43d52052af2d\") " Oct 30 19:24:13.790931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.790900 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca" pod "d648c22d-6c5b-47f4-b7b1-43d52052af2d" (UID: "d648c22d-6c5b-47f4-b7b1-43d52052af2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:13.791274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.791252 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:13.792668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.792635 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca" pod "d648c22d-6c5b-47f4-b7b1-43d52052af2d" (UID: "d648c22d-6c5b-47f4-b7b1-43d52052af2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:13.792793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.792774 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w" pod "d648c22d-6c5b-47f4-b7b1-43d52052af2d" (UID: "d648c22d-6c5b-47f4-b7b1-43d52052af2d"). InnerVolumeSpecName "kube-api-access-6zd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:13.869697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.869656 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:13.893345 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.893297 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d648c22d-6c5b-47f4-b7b1-43d52052af2d-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:13.893345 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:13.893341 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zd7w\" (UniqueName: \"kubernetes.io/projected/d648c22d-6c5b-47f4-b7b1-43d52052af2d-kube-api-access-6zd7w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:14.124797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:14.124760 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:14.506357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:14.506325 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" Oct 30 19:24:14.506530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:14.506365 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj" event={"ID":"d648c22d-6c5b-47f4-b7b1-43d52052af2d","Type":"ContainerDied","Data":"c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3"} Oct 30 19:24:14.506530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:14.506426 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7cf4d5209579c3fd2ad5838c9665fc0ca76fc19e83407cc47fd0e7ade2974b3" Oct 30 19:24:15.551976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:15.551939 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:17.136645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:17.136604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:18.812113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.812045 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:24:18.819935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.819892 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d648c22d-6c5b-47f4-b7b1-43d52052af2d" containerName="revoke-keys" Oct 30 19:24:18.820197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.820175 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d648c22d-6c5b-47f4-b7b1-43d52052af2d" containerName="revoke-keys" Oct 30 19:24:18.820519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.820502 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d648c22d-6c5b-47f4-b7b1-43d52052af2d" containerName="revoke-keys" Oct 30 19:24:18.840765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.840727 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:24:18.849953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:18.841692 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.002931 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.003304 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.003452 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.003586 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jbr\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.003709 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.003869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.003853 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.004064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.004045 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.004298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.004279 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.004435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.004419 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.122880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.122766 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.123233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.123194 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.123462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.123446 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.123646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.123618 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.124958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.124937 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.125240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.125211 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jbr\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.126049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.126022 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.126650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.124647 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.127639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.124291 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.127798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.126450 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.128018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.125114 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.128183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.127554 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.128395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.126615 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.129050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.128711 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.164890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.159761 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.166376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.166349 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.181144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.172171 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.181144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.174519 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jbr\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\") pod \"maas-default-gateway-openshift-default-6f54779779-scx8v\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.225018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.224972 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:19.552600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.552569 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:24:19.556468 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:19.556442 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd6d3e4_dc25_49d8_8d14_155f82e0b3b6.slice/crio-387b00d21821f7cf312fd720e5bcbbb16ba396f566282894640359d0014b950f WatchSource:0}: Error finding container 387b00d21821f7cf312fd720e5bcbbb16ba396f566282894640359d0014b950f: Status 404 returned error can't find the container with id 387b00d21821f7cf312fd720e5bcbbb16ba396f566282894640359d0014b950f Oct 30 19:24:19.558884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.558850 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:19.558970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.558914 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:19.558970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.558944 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:19.579501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.579459 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" event={"ID":"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6","Type":"ContainerStarted","Data":"387b00d21821f7cf312fd720e5bcbbb16ba396f566282894640359d0014b950f"} Oct 30 19:24:19.647323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:19.647285 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:20.608112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:20.608054 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" event={"ID":"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6","Type":"ContainerStarted","Data":"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a"} Oct 30 19:24:20.702237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:20.702176 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podStartSLOduration=2.702159964 podStartE2EDuration="2.702159964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:20.680555279 +0000 UTC m=+1168.392080671" watchObservedRunningTime="2026-10-30 19:24:20.702159964 +0000 UTC m=+1168.413685330" Oct 30 19:24:21.029462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.029382 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn"] Oct 30 19:24:21.033515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.033492 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.049119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.049084 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:24:21.049256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.049230 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:24:21.053045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.053019 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn"] Oct 30 19:24:21.157192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.157154 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.157356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.157234 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.157398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.157374 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjsbg\" (UniqueName: \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.225262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.225229 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:21.226849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.226823 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:21.226971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.226879 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:21.258041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.258007 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjsbg\" (UniqueName: \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.258220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.258086 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.258220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.258179 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.258953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.258927 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.285156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.279444 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.307116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.300273 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjsbg\" (UniqueName: \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\") pod \"maas-api-revoke-sub-021e8643-x94rn\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.347723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.347695 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:21.609327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.609285 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:21.703614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:21.702934 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn"] Oct 30 19:24:21.707062 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:21.707023 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f12094_d639_46a9_a531_6c31d5e9b830.slice/crio-64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04 WatchSource:0}: Error finding container 64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04: Status 404 returned error can't find the container with id 64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04 Oct 30 19:24:22.226523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:22.226487 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:22.226650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:22.226540 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:22.626519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:22.626476 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" event={"ID":"c5f12094-d639-46a9-a531-6c31d5e9b830","Type":"ContainerStarted","Data":"69acea9fc9d6af41e16802a006cd33aa2a14b6740bf539ad5716c93409a7c7e7"} Oct 30 19:24:22.627067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:22.626544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" event={"ID":"c5f12094-d639-46a9-a531-6c31d5e9b830","Type":"ContainerStarted","Data":"64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04"} Oct 30 19:24:22.662972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:22.662921 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" podStartSLOduration=1.6629058780000001 podStartE2EDuration="1.662905878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:22.659848302 +0000 UTC m=+1170.371373668" watchObservedRunningTime="2026-10-30 19:24:22.662905878 +0000 UTC m=+1170.374431244" Oct 30 19:24:23.225370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:23.225336 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:23.225575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:23.225395 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:23.395116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:23.392026 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:23.634364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:23.634331 4990 generic.go:356] "Generic (PLEG): container finished" podID="c5f12094-d639-46a9-a531-6c31d5e9b830" containerID="69acea9fc9d6af41e16802a006cd33aa2a14b6740bf539ad5716c93409a7c7e7" exitCode=0 Oct 30 19:24:23.634816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:23.634410 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" event={"ID":"c5f12094-d639-46a9-a531-6c31d5e9b830","Type":"ContainerDied","Data":"69acea9fc9d6af41e16802a006cd33aa2a14b6740bf539ad5716c93409a7c7e7"} Oct 30 19:24:24.226192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:24.226150 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:24.226379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:24.226229 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:24.853597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:24.852252 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:25.031130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.028056 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\") pod \"c5f12094-d639-46a9-a531-6c31d5e9b830\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " Oct 30 19:24:25.031130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.028291 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\" (UniqueName: \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\") pod \"c5f12094-d639-46a9-a531-6c31d5e9b830\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " Oct 30 19:24:25.031130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.028360 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\") pod \"c5f12094-d639-46a9-a531-6c31d5e9b830\" (UID: \"c5f12094-d639-46a9-a531-6c31d5e9b830\") " Oct 30 19:24:25.031130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.029364 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca" pod "c5f12094-d639-46a9-a531-6c31d5e9b830" (UID: "c5f12094-d639-46a9-a531-6c31d5e9b830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:25.032174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.032104 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca" pod "c5f12094-d639-46a9-a531-6c31d5e9b830" (UID: "c5f12094-d639-46a9-a531-6c31d5e9b830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:25.032741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.032711 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg" pod "c5f12094-d639-46a9-a531-6c31d5e9b830" (UID: "c5f12094-d639-46a9-a531-6c31d5e9b830"). InnerVolumeSpecName "kube-api-access-bjsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:25.130284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.130196 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:25.130284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.130227 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjsbg\" (UniqueName: \"kubernetes.io/projected/c5f12094-d639-46a9-a531-6c31d5e9b830-kube-api-access-bjsbg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:25.130284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.130240 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f12094-d639-46a9-a531-6c31d5e9b830-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:25.225686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.225645 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:25.225879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.225724 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:25.294724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.294675 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:25.653449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.653415 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" Oct 30 19:24:25.653449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.653431 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn" event={"ID":"c5f12094-d639-46a9-a531-6c31d5e9b830","Type":"ContainerDied","Data":"64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04"} Oct 30 19:24:25.653661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.653477 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f87ae897b145103033285cb8e3f5448bb4e9d8d037ef41de317d4efc12eb04" Oct 30 19:24:25.812480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.812448 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk"] Oct 30 19:24:25.812971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.812957 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c5f12094-d639-46a9-a531-6c31d5e9b830" containerName="revoke-keys" Oct 30 19:24:25.813023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.812972 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f12094-d639-46a9-a531-6c31d5e9b830" containerName="revoke-keys" Oct 30 19:24:25.813083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.813074 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f12094-d639-46a9-a531-6c31d5e9b830" containerName="revoke-keys" Oct 30 19:24:25.816470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.816454 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:25.828743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.828706 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:24:25.828865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.828844 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:24:25.836933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.836905 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk"] Oct 30 19:24:25.939191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.939062 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:25.939675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.939263 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:25.939675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:25.939317 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv258\" (UniqueName: \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.041238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.041186 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.041419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.041387 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.041473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.041453 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv258\" (UniqueName: \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.042198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.042173 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.043539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.043520 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.059287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.059258 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv258\" (UniqueName: \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\") pod \"maas-api-revoke-sub-3fb1bfe6-cqdxk\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.132023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.131801 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:26.226054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.225568 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:26.226054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.225641 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:26.436389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.436324 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk"] Oct 30 19:24:26.440377 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:26.440343 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a5706b_b468_428b_bdfd_92ee5faf2055.slice/crio-399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502 WatchSource:0}: Error finding container 399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502: Status 404 returned error can't find the container with id 399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502 Oct 30 19:24:26.662187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:26.662145 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" event={"ID":"39a5706b-b468-428b-bdfd-92ee5faf2055","Type":"ContainerStarted","Data":"399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502"} Oct 30 19:24:27.128618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:27.128583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:27.225810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:27.225765 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:24:27.226012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:27.225834 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 19:24:27.673758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:27.673662 4990 generic.go:356] "Generic (PLEG): container finished" podID="39a5706b-b468-428b-bdfd-92ee5faf2055" containerID="d463b78bba4e76b3356e15a54ff3691b8a4a0e492456ff2d652267c5aea4af65" exitCode=0 Oct 30 19:24:27.673947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:27.673764 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" event={"ID":"39a5706b-b468-428b-bdfd-92ee5faf2055","Type":"ContainerDied","Data":"d463b78bba4e76b3356e15a54ff3691b8a4a0e492456ff2d652267c5aea4af65"} Oct 30 19:24:28.229780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.229750 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:28.230184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.230019 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:28.230755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.230736 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:24:28.865140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.865114 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:28.879810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.879773 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\") pod \"39a5706b-b468-428b-bdfd-92ee5faf2055\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " Oct 30 19:24:28.879973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.879823 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\") pod \"39a5706b-b468-428b-bdfd-92ee5faf2055\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " Oct 30 19:24:28.880031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.879964 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\" (UniqueName: \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\") pod \"39a5706b-b468-428b-bdfd-92ee5faf2055\" (UID: \"39a5706b-b468-428b-bdfd-92ee5faf2055\") " Oct 30 19:24:28.880196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.880173 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca" pod "39a5706b-b468-428b-bdfd-92ee5faf2055" (UID: "39a5706b-b468-428b-bdfd-92ee5faf2055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:28.880547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.880524 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:28.881946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.881919 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258" pod "39a5706b-b468-428b-bdfd-92ee5faf2055" (UID: "39a5706b-b468-428b-bdfd-92ee5faf2055"). InnerVolumeSpecName "kube-api-access-zv258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:28.881946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.881936 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca" pod "39a5706b-b468-428b-bdfd-92ee5faf2055" (UID: "39a5706b-b468-428b-bdfd-92ee5faf2055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:28.981987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.981882 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a5706b-b468-428b-bdfd-92ee5faf2055-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:28.981987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:28.981917 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv258\" (UniqueName: \"kubernetes.io/projected/39a5706b-b468-428b-bdfd-92ee5faf2055-kube-api-access-zv258\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:24:29.018008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:29.017929 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:29.691240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:29.691208 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" Oct 30 19:24:29.691749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:29.691234 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk" event={"ID":"39a5706b-b468-428b-bdfd-92ee5faf2055","Type":"ContainerDied","Data":"399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502"} Oct 30 19:24:29.691749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:29.691284 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399831025bd2d6abbe8ddb4f188fece2bd20fca72714ba0750aed929557c7502" Oct 30 19:24:30.335105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.335055 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:24:30.335944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.335924 4990 cpu_manager.go:414] "containerMap: removing container" podUID="39a5706b-b468-428b-bdfd-92ee5faf2055" containerName="revoke-keys" Oct 30 19:24:30.335998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.335948 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a5706b-b468-428b-bdfd-92ee5faf2055" containerName="revoke-keys" Oct 30 19:24:30.336103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.336075 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a5706b-b468-428b-bdfd-92ee5faf2055" containerName="revoke-keys" Oct 30 19:24:30.339999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.339980 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.363712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.363677 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:24:30.402736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.402698 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.402925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.402753 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.402925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.402884 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.403064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.402926 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.403064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.402951 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.403232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.403066 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.403232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.403137 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4hn\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.403232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.403188 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.404568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.403235 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504299 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504339 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504381 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504401 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504416 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.504598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504507 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504542 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4hn\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.505166 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504873 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.505174 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.505224 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.504772 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.505273 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.505564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.505477 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.506914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.506889 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.507261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.507239 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.521850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.521817 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4hn\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.522387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.522363 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.654600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.654514 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:30.988534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:30.988339 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:24:31.002544 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:31.002352 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713fa0d3_f937_42e1_a3a7_fab259226bc2.slice/crio-79f8c24508e2b98a37a2b52f0b43781a8c3a80def5d7144d7061d0210ea12ec2 WatchSource:0}: Error finding container 79f8c24508e2b98a37a2b52f0b43781a8c3a80def5d7144d7061d0210ea12ec2: Status 404 returned error can't find the container with id 79f8c24508e2b98a37a2b52f0b43781a8c3a80def5d7144d7061d0210ea12ec2 Oct 30 19:24:31.009773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.009724 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:31.009916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.009830 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:31.009916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.009902 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:31.194592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.194510 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:31.731116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.730656 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" event={"ID":"713fa0d3-f937-42e1-a3a7-fab259226bc2","Type":"ContainerStarted","Data":"72c1b2d5ef435977893d1caea66928bb77e6f928dc9e9efa016020ac8a6fa01e"} Oct 30 19:24:31.731116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.730721 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" event={"ID":"713fa0d3-f937-42e1-a3a7-fab259226bc2","Type":"ContainerStarted","Data":"79f8c24508e2b98a37a2b52f0b43781a8c3a80def5d7144d7061d0210ea12ec2"} Oct 30 19:24:31.805921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:31.805855 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podStartSLOduration=1.80583636 podStartE2EDuration="1.80583636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:31.791973912 +0000 UTC m=+1179.503499277" watchObservedRunningTime="2026-10-30 19:24:31.80583636 +0000 UTC m=+1179.517361730" Oct 30 19:24:32.655566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:32.655535 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:32.657140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:32.657113 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:32.657251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:32.657162 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:33.032865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:33.032768 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:33.654975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:33.654931 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:33.655190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:33.654995 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:34.655375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:34.655299 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:34.655375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:34.655353 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:34.737993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:34.737958 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.115140 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:24:35.115085 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:24:35.493166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.492130 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 19:24:35.499527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.499502 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.510118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.508897 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 19:24:35.523350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.523295 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"distinct-w0-c5449c-kserve-self-signed-certs\"" Oct 30 19:24:35.602503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.602437 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.633791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.633762 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.633980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.633823 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.633980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.633888 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.634157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.634031 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.634157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.634105 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nrd\" (UniqueName: \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.634244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.634208 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.655226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.655195 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:35.655346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.655247 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:35.735987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.735947 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736004 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736071 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736203 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736263 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nrd\" (UniqueName: \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736376 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.736585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.736492 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.737145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.737119 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.737575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.737546 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.742617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.742596 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.743553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.743487 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.755619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.754355 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nrd\" (UniqueName: \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\") pod \"distinct-w0-c5449c-kserve-5c769f964f-8tv8h\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:35.822798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:35.822759 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:36.158924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.158899 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 19:24:36.161029 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:36.160998 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e0aa01_2f6b_4d1c_ae2e_9644e0d7bb23.slice/crio-28340b739ac929aebb461310c493bc989282cb93a11086ba02050fd93180b5d6 WatchSource:0}: Error finding container 28340b739ac929aebb461310c493bc989282cb93a11086ba02050fd93180b5d6: Status 404 returned error can't find the container with id 28340b739ac929aebb461310c493bc989282cb93a11086ba02050fd93180b5d6 Oct 30 19:24:36.413156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.413049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:36.655733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.655693 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:36.655942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.655751 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:36.806497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.806455 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" event={"ID":"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23","Type":"ContainerStarted","Data":"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396"} Oct 30 19:24:36.806974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.806506 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" event={"ID":"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23","Type":"ContainerStarted","Data":"28340b739ac929aebb461310c493bc989282cb93a11086ba02050fd93180b5d6"} Oct 30 19:24:36.822982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.822956 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:36.842657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:36.842595 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podStartSLOduration=1.84257001 podStartE2EDuration="1.84257001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:36.841041062 +0000 UTC m=+1184.552566429" watchObservedRunningTime="2026-10-30 19:24:36.84257001 +0000 UTC m=+1184.554095377" Oct 30 19:24:37.655747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:37.655706 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:37.655917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:37.655782 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 19:24:38.205553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:38.205499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:38.660499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:38.660459 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:38.660871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:38.660846 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:38.661677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:38.661653 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:24:39.834560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:39.834524 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:41.510715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:41.510681 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:44.304478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:44.304435 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:45.226506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.226472 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:24:45.230408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.230389 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.251325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.251291 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:24:45.369759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.369714 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.369759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.369753 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.369782 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.369929 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.369997 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.370016 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.370155 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.370188 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.370354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.370213 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qdd\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471121 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471184 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471214 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qdd\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471307 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471333 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471374 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471517 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471610 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471617 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471638 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.471836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471754 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.472375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.471895 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.472375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.472188 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.472375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.472353 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.474330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.474302 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.474594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.474570 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.487689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.487628 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.487855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.487832 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qdd\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.545887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.545860 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:45.801011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.800971 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:24:45.806074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.806049 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.823997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.823957 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:45.828150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.828121 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:24:45.842426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.842402 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.880770 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.880867 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.880909 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881036 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881124 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881157 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvxk\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881210 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881243 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.882923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.881272 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.894301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.894278 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:24:45.895375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.895337 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:45.895491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.895413 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:45.895491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.895482 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:45.908430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.908407 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:24:45.983475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983217 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983325 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983362 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983499 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983564 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983588 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvxk\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983642 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983677 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.983758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.983702 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.985221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.984895 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.985221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.985171 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.985448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.985349 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.986015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.985987 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.986376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.986345 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.991139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.989696 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:45.991139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:45.989848 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:46.004106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.002237 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:46.004106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.003354 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvxk\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:46.018557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.018521 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:46.124874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.124791 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:46.456463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.454792 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:24:46.459229 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:46.459197 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906cda27_24c5_413b_8166_ea2439390411.slice/crio-d40cee6d2c85fa77aca48a1b90043b7dab7da22c400ba8033e7da00c82f90929 WatchSource:0}: Error finding container d40cee6d2c85fa77aca48a1b90043b7dab7da22c400ba8033e7da00c82f90929: Status 404 returned error can't find the container with id d40cee6d2c85fa77aca48a1b90043b7dab7da22c400ba8033e7da00c82f90929 Oct 30 19:24:46.461805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.461769 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:46.461880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.461839 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:46.461880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.461871 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:46.549255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.549229 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:46.915144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.915077 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" event={"ID":"bbb7b181-fefc-4d27-8ee3-1188824ee897","Type":"ContainerStarted","Data":"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147"} Oct 30 19:24:46.915352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.915164 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" event={"ID":"bbb7b181-fefc-4d27-8ee3-1188824ee897","Type":"ContainerStarted","Data":"31499956928e2bd3ada76eb03bc2610b4bba9f5cf27499d3b44e7b0889e93967"} Oct 30 19:24:46.918755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.918632 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" event={"ID":"906cda27-24c5-413b-8166-ea2439390411","Type":"ContainerStarted","Data":"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752"} Oct 30 19:24:46.918755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.918699 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" event={"ID":"906cda27-24c5-413b-8166-ea2439390411","Type":"ContainerStarted","Data":"d40cee6d2c85fa77aca48a1b90043b7dab7da22c400ba8033e7da00c82f90929"} Oct 30 19:24:46.956719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:46.956660 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podStartSLOduration=1.95664086 podStartE2EDuration="1.95664086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.952702937 +0000 UTC m=+1194.664228304" watchObservedRunningTime="2026-10-30 19:24:46.95664086 +0000 UTC m=+1194.668166229" Oct 30 19:24:47.125176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.125134 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:47.126515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.126488 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:47.126672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.126545 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:47.546154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.546115 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:47.548177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.548146 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:47.548251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:47.548208 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:48.030680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:48.030637 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:48.127137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:48.125664 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:48.127137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:48.125730 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:48.546702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:48.546666 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:48.547172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:48.546718 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:49.126114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:49.125662 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:49.126114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:49.125744 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:49.546718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:49.546631 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:49.546718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:49.546682 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:49.842734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:49.842696 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.137175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:50.136439 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:50.137175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:50.136508 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:50.547281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:50.547197 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:50.547281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:50.547258 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:51.125922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:51.125880 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:51.126144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:51.125953 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:51.547145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:51.547028 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:51.547145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:51.547086 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:51.648882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:51.648843 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:52.126104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.126042 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:52.126317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.126163 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:52.546324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.546235 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:52.546324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.546291 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 19:24:52.557904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.557866 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:24:52.558659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.558636 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:24:52.596912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.596878 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:24:52.597494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:52.597476 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:24:53.125599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.125552 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:53.125820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.125610 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 19:24:53.538813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.538732 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:53.573006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.572970 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:53.574045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.574014 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:53.578454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.578429 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:24:53.633883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:53.633827 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podStartSLOduration=8.633813745 podStartE2EDuration="8.633813745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:47.003461681 +0000 UTC m=+1194.714987050" watchObservedRunningTime="2026-10-30 19:24:53.633813745 +0000 UTC m=+1201.345339110" Oct 30 19:24:54.130315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:54.130284 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:54.130586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:54.130562 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:54.131363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:54.131341 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:24:55.294758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.294709 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:24:55.313787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.309766 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.313787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.311780 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:55.332997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.332963 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:24:55.370862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.370827 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.370952 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371078 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371148 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371213 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371244 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371368 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371506 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqs4r\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.371567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.371540 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472469 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472591 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472671 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472700 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472730 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472756 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.472929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472834 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.472991 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqs4r\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.473053 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.473356 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.473356 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.473390 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.473813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.473736 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.474086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.474056 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.475303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.475284 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.476596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.476579 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.490207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.490185 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.490980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.490952 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqs4r\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.636419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.636376 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:55.964077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.964046 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:24:55.966166 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:24:55.966137 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63745ae5_9ae5_4fcb_9b2b_74d6d19ac0a6.slice/crio-da0cbee09e15a801ec30c34c931cd8133fef8bf1dbbee5c84636d4289e5b5378 WatchSource:0}: Error finding container da0cbee09e15a801ec30c34c931cd8133fef8bf1dbbee5c84636d4289e5b5378: Status 404 returned error can't find the container with id da0cbee09e15a801ec30c34c931cd8133fef8bf1dbbee5c84636d4289e5b5378 Oct 30 19:24:55.968725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.968690 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:55.968831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.968764 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:55.968831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:55.968793 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:24:56.013482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:56.013454 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" event={"ID":"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6","Type":"ContainerStarted","Data":"da0cbee09e15a801ec30c34c931cd8133fef8bf1dbbee5c84636d4289e5b5378"} Oct 30 19:24:57.012164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.012128 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:57.031470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.031433 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" event={"ID":"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6","Type":"ContainerStarted","Data":"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15"} Oct 30 19:24:57.099184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.092992 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podStartSLOduration=2.092971077 podStartE2EDuration="2.092971077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:57.077572212 +0000 UTC m=+1204.789097581" watchObservedRunningTime="2026-10-30 19:24:57.092971077 +0000 UTC m=+1204.804496444" Oct 30 19:24:57.233460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.233423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:57.636505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.636468 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:24:57.638398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.638368 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:24:57.638529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:57.638425 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:24:58.180115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:58.179903 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:58.637268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:58.637229 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:24:58.637488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:58.637291 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:24:58.784841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:58.784800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:59.231228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:59.231166 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:59.642555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:59.642515 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:24:59.642854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:24:59.642822 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:25:00.637424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:00.637389 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:25:00.637858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:00.637453 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:25:00.812380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:00.812339 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.636960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:01.636926 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:25:01.637152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:01.636990 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:25:02.409669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:02.409634 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:02.637728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:02.637695 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:25:02.637939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:02.637761 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:25:03.641529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:03.641499 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:25:03.641925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:03.641902 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:25:03.642514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:03.642497 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:25:04.139469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:04.139415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:05.099528 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:25:05.099480 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:25:05.935980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:05.935943 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:07.617300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:07.617264 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:09.510117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:09.510043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:11.212300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:11.212244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:12.817588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:12.817545 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:13.615576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.615540 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 19:25:13.619799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.619773 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.645518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.645493 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"distinct-2-w0-c5449c-kserve-self-signed-certs\"" Oct 30 19:25:13.650173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.650147 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 19:25:13.663843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.663801 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qx7m\" (UniqueName: \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.664017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.663862 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.664017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.663959 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.664017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.664012 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.664531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.664140 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.664531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.664243 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.719165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.719126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:13.765590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765560 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qx7m\" (UniqueName: \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.765768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765608 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.765768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765729 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.765861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765774 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.765861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765809 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.765965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.765895 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.766236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.766213 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.766384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.766264 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.766384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.766331 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.768153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.768130 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.768385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.768369 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.782751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.782727 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qx7m\" (UniqueName: \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\") pod \"distinct-2-w0-c5449c-kserve-fff65f6db-67sqt\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:13.937226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:13.937131 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:14.242559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:14.242535 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 19:25:14.244742 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:25:14.244697 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc0847c_9475_4fd2_8184_1df73e421707.slice/crio-0b14fb951d7942ad469308f46aa8b554316fba36efc1a587aed3638c4f5cc750 WatchSource:0}: Error finding container 0b14fb951d7942ad469308f46aa8b554316fba36efc1a587aed3638c4f5cc750: Status 404 returned error can't find the container with id 0b14fb951d7942ad469308f46aa8b554316fba36efc1a587aed3638c4f5cc750 Oct 30 19:25:14.611277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:14.611224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:15.214708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:15.214665 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" event={"ID":"0fc0847c-9475-4fd2-8184-1df73e421707","Type":"ContainerStarted","Data":"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70"} Oct 30 19:25:15.214708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:15.214715 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" event={"ID":"0fc0847c-9475-4fd2-8184-1df73e421707","Type":"ContainerStarted","Data":"0b14fb951d7942ad469308f46aa8b554316fba36efc1a587aed3638c4f5cc750"} Oct 30 19:25:15.250251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:15.250198 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podStartSLOduration=2.2501832569999998 podStartE2EDuration="2.250183257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:15.248808513 +0000 UTC m=+1222.960333883" watchObservedRunningTime="2026-10-30 19:25:15.250183257 +0000 UTC m=+1222.961708623" Oct 30 19:25:15.938182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:15.938152 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:16.263938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:16.263830 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:17.837728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:17.832524 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:19.730072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:19.730031 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:21.312165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:21.312086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:23.108515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:23.108476 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:23.937976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:23.937942 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:23.950461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:23.950437 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:24.299217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:24.299126 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:25:24.811968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:24.811742 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.427827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:26.427788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:28.238400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:28.238357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.354275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.354243 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 19:25:30.360122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.360081 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.398189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.398154 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 19:25:30.495683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.495641 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.531994 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532052 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532124 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532213 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532237 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89ng\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532279 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532330 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532374 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.534144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.532483 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633206 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633241 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89ng\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633274 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633315 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633350 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633437 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633542 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633565 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633619 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633710 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.633962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.633855 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.634127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.634080 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.634351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.634326 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.634439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.634416 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.635991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.635965 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.636383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.636361 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.650062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.650038 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89ng\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.650183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.650164 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:30.679579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:30.679550 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:31.014553 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:25:31.014516 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba5e617_1e5c_47bc_94ed_0a9937f4c1da.slice/crio-5633f6c4153856f866f0d587cbf456f306c79dfa632c6ba5679541088760fd2d WatchSource:0}: Error finding container 5633f6c4153856f866f0d587cbf456f306c79dfa632c6ba5679541088760fd2d: Status 404 returned error can't find the container with id 5633f6c4153856f866f0d587cbf456f306c79dfa632c6ba5679541088760fd2d Oct 30 19:25:31.014678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.014546 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 19:25:31.017037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.017005 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:25:31.017106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.017074 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:25:31.017148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.017120 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:25:31.367132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.367075 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" event={"ID":"eba5e617-1e5c-47bc-94ed-0a9937f4c1da","Type":"ContainerStarted","Data":"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6"} Oct 30 19:25:31.367543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.367156 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" event={"ID":"eba5e617-1e5c-47bc-94ed-0a9937f4c1da","Type":"ContainerStarted","Data":"5633f6c4153856f866f0d587cbf456f306c79dfa632c6ba5679541088760fd2d"} Oct 30 19:25:31.408840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.408777 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podStartSLOduration=1.40875818 podStartE2EDuration="1.40875818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:31.404910742 +0000 UTC m=+1239.116436109" watchObservedRunningTime="2026-10-30 19:25:31.40875818 +0000 UTC m=+1239.120283548" Oct 30 19:25:31.683183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:31.680547 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:32.449592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:32.449556 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:32.683233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:32.683198 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:32.683482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:32.683452 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:33.684115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:33.683618 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:33.684115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:33.683679 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:33.946597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:33.946503 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.686114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:34.683454 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:34.686114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:34.683536 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:34.727859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:34.727805 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.125017 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:25:35.124982 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:25:35.683622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:35.683178 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:35.683622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:35.683240 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:36.680452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:36.680415 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:36.681018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:36.680988 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:36.717291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:36.717251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.679961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:37.679913 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:37.680180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:37.679989 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 19:25:38.473698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:38.473663 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.685079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:38.685045 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:38.685466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:38.685449 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:38.686143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:38.686118 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 19:25:40.218675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:40.218638 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.049570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:42.049529 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:43.748050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.748004 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p"] Oct 30 19:25:43.778190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.778158 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:43.821126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.812532 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:25:43.821126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.812666 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:25:43.890215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.890166 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p"] Oct 30 19:25:43.969153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.922186 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:25:43.969153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.947775 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:25:43.970958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.970929 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:43.971261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.971238 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:43.971611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:43.971580 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nnx\" (UniqueName: \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.077267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.077222 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:25:44.134116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.107982 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nnx\" (UniqueName: \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.134116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.108133 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.134116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.108225 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.173533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.135689 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.173533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.143981 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.223624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.223514 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nnx\" (UniqueName: \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\") pod \"maas-api-revoke-sub-8d211bdf-fdx6p\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.500161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.499405 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:44.597470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.597182 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:44.832104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:44.832059 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p"] Oct 30 19:25:44.834387 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:25:44.834353 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa843b4_abd5_48e0_af05_5d55747e6322.slice/crio-86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51 WatchSource:0}: Error finding container 86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51: Status 404 returned error can't find the container with id 86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51 Oct 30 19:25:45.569378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:45.569331 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" event={"ID":"3fa843b4-abd5-48e0-af05-5d55747e6322","Type":"ContainerStarted","Data":"2ab49e4042b02fd608a232dbe38a996e49e0ca128a98e354a1fea69e7c5eb0ca"} Oct 30 19:25:45.569545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:45.569400 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" event={"ID":"3fa843b4-abd5-48e0-af05-5d55747e6322","Type":"ContainerStarted","Data":"86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51"} Oct 30 19:25:45.607257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:45.607211 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" podStartSLOduration=2.6071954550000003 podStartE2EDuration="2.607195455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:45.603718902 +0000 UTC m=+1253.315244268" watchObservedRunningTime="2026-10-30 19:25:45.607195455 +0000 UTC m=+1253.318720820" Oct 30 19:25:46.580206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:46.580169 4990 generic.go:356] "Generic (PLEG): container finished" podID="3fa843b4-abd5-48e0-af05-5d55747e6322" containerID="2ab49e4042b02fd608a232dbe38a996e49e0ca128a98e354a1fea69e7c5eb0ca" exitCode=0 Oct 30 19:25:46.580633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:46.580218 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" event={"ID":"3fa843b4-abd5-48e0-af05-5d55747e6322","Type":"ContainerDied","Data":"2ab49e4042b02fd608a232dbe38a996e49e0ca128a98e354a1fea69e7c5eb0ca"} Oct 30 19:25:46.605240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:46.605211 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:47.760785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.760759 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:47.873803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.873705 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\") pod \"3fa843b4-abd5-48e0-af05-5d55747e6322\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " Oct 30 19:25:47.873983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.873816 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\" (UniqueName: \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\") pod \"3fa843b4-abd5-48e0-af05-5d55747e6322\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " Oct 30 19:25:47.874158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.874029 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\") pod \"3fa843b4-abd5-48e0-af05-5d55747e6322\" (UID: \"3fa843b4-abd5-48e0-af05-5d55747e6322\") " Oct 30 19:25:47.874838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.874803 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca" pod "3fa843b4-abd5-48e0-af05-5d55747e6322" (UID: "3fa843b4-abd5-48e0-af05-5d55747e6322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:47.876042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.876018 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca" pod "3fa843b4-abd5-48e0-af05-5d55747e6322" (UID: "3fa843b4-abd5-48e0-af05-5d55747e6322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:47.877060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.877031 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx" pod "3fa843b4-abd5-48e0-af05-5d55747e6322" (UID: "3fa843b4-abd5-48e0-af05-5d55747e6322"). InnerVolumeSpecName "kube-api-access-l4nnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:47.975671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.975635 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:25:47.975671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.975664 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa843b4-abd5-48e0-af05-5d55747e6322-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:25:47.975671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:47.975674 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4nnx\" (UniqueName: \"kubernetes.io/projected/3fa843b4-abd5-48e0-af05-5d55747e6322-kube-api-access-l4nnx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:25:48.334387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:48.334345 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:48.599791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:48.599713 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" Oct 30 19:25:48.599791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:48.599743 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p" event={"ID":"3fa843b4-abd5-48e0-af05-5d55747e6322","Type":"ContainerDied","Data":"86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51"} Oct 30 19:25:48.599991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:48.599806 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f14e2ec92c9cdcab46d95ed8460ff053f6351d1db0ca709bdee3d21304bd51" Oct 30 19:25:49.907035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:49.906988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:51.611790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:51.611754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:53.304123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:53.304038 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:54.339470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.339434 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 19:25:54.340327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.340304 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3fa843b4-abd5-48e0-af05-5d55747e6322" containerName="revoke-keys" Oct 30 19:25:54.340327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.340325 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa843b4-abd5-48e0-af05-5d55747e6322" containerName="revoke-keys" Oct 30 19:25:54.340505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.340471 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa843b4-abd5-48e0-af05-5d55747e6322" containerName="revoke-keys" Oct 30 19:25:54.344534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.344506 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.372917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.372880 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"distinct-2-w4-b97379-kserve-self-signed-certs\"" Oct 30 19:25:54.379103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.379063 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 19:25:54.455378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.455334 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:54.467703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.467670 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnr66\" (UniqueName: \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.467868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.467821 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.467978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.467959 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.468038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.468018 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.468128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.468066 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.468237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.468210 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570109 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570183 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570283 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570370 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnr66\" (UniqueName: \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570463 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570562 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570577 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570629 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.570738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.570688 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.573159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.573128 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.573316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.573260 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.587959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.587901 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnr66\" (UniqueName: \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\") pod \"distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.660124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.659970 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:54.987186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:54.987154 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 19:25:54.989005 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:25:54.988965 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0746b3fc_6eba_4f5f_bfee_2dde967613f6.slice/crio-432350c3ae033e01a08e682bae48b2a5c91d5e9f37b3e662a5b3da67b30ce74e WatchSource:0}: Error finding container 432350c3ae033e01a08e682bae48b2a5c91d5e9f37b3e662a5b3da67b30ce74e: Status 404 returned error can't find the container with id 432350c3ae033e01a08e682bae48b2a5c91d5e9f37b3e662a5b3da67b30ce74e Oct 30 19:25:55.091068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:55.091022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.655703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:55.655664 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" event={"ID":"0746b3fc-6eba-4f5f-bfee-2dde967613f6","Type":"ContainerStarted","Data":"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e"} Oct 30 19:25:55.656139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:55.655719 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" event={"ID":"0746b3fc-6eba-4f5f-bfee-2dde967613f6","Type":"ContainerStarted","Data":"432350c3ae033e01a08e682bae48b2a5c91d5e9f37b3e662a5b3da67b30ce74e"} Oct 30 19:25:55.661039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:55.661017 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:25:55.690071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:55.690025 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podStartSLOduration=1.690012643 podStartE2EDuration="1.690012643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:55.689241303 +0000 UTC m=+1263.400766668" watchObservedRunningTime="2026-10-30 19:25:55.690012643 +0000 UTC m=+1263.401538008" Oct 30 19:25:56.737005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:56.736966 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.904406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:56.904363 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 19:25:58.290451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:58.290407 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:58.434141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:25:58.434073 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.019317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.019282 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.025959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.025932 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p"] Oct 30 19:26:00.030473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.030452 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.051647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.051614 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.051819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.051702 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46blr\" (UniqueName: \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.051890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.051868 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.055222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.055189 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:00.055349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.055282 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:00.059768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.059743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p"] Oct 30 19:26:00.154618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.154577 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.154983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.154960 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.155190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.155172 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46blr\" (UniqueName: \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.156567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.156541 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.164031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.159610 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.173180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.173155 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46blr\" (UniqueName: \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\") pod \"maas-api-revoke-sub-252cee94-6455p\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.350589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.350555 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:00.665986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.665951 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p"] Oct 30 19:26:00.668596 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:00.668564 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cec7ba_2d6e_43ee_a612_857a76f8c08a.slice/crio-76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0 WatchSource:0}: Error finding container 76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0: Status 404 returned error can't find the container with id 76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0 Oct 30 19:26:00.701505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:00.701466 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" event={"ID":"42cec7ba-2d6e-43ee-a612-857a76f8c08a","Type":"ContainerStarted","Data":"76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0"} Oct 30 19:26:01.719021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:01.718980 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:01.720036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:01.719785 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" event={"ID":"42cec7ba-2d6e-43ee-a612-857a76f8c08a","Type":"ContainerStarted","Data":"71a30857ed8e0ebe179ed218a84245ee27ce6fe2982f3bf0ad5de5a8f8ed9173"} Oct 30 19:26:01.757314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:01.757250 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" podStartSLOduration=1.757230786 podStartE2EDuration="1.757230786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:01.756144836 +0000 UTC m=+1269.467670202" watchObservedRunningTime="2026-10-30 19:26:01.757230786 +0000 UTC m=+1269.468756152" Oct 30 19:26:02.735253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:02.735212 4990 generic.go:356] "Generic (PLEG): container finished" podID="42cec7ba-2d6e-43ee-a612-857a76f8c08a" containerID="71a30857ed8e0ebe179ed218a84245ee27ce6fe2982f3bf0ad5de5a8f8ed9173" exitCode=0 Oct 30 19:26:02.736357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:02.736143 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" event={"ID":"42cec7ba-2d6e-43ee-a612-857a76f8c08a","Type":"ContainerDied","Data":"71a30857ed8e0ebe179ed218a84245ee27ce6fe2982f3bf0ad5de5a8f8ed9173"} Oct 30 19:26:04.362340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.362302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.404689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.396419 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:04.736319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.736151 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:04.784570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.774208 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:26:04.784570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.774272 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:04.806156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.805728 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:04.825785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.825759 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:04.842992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.842962 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:04.961944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.961870 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\") pod \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " Oct 30 19:26:04.962129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.962044 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\" (UniqueName: \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\") pod \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " Oct 30 19:26:04.962208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.962128 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\") pod \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\" (UID: \"42cec7ba-2d6e-43ee-a612-857a76f8c08a\") " Oct 30 19:26:04.962324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.962296 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca" pod "42cec7ba-2d6e-43ee-a612-857a76f8c08a" (UID: "42cec7ba-2d6e-43ee-a612-857a76f8c08a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:04.962709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.962682 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:04.964194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.964164 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr" pod "42cec7ba-2d6e-43ee-a612-857a76f8c08a" (UID: "42cec7ba-2d6e-43ee-a612-857a76f8c08a"). InnerVolumeSpecName "kube-api-access-46blr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:04.964295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:04.964214 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca" pod "42cec7ba-2d6e-43ee-a612-857a76f8c08a" (UID: "42cec7ba-2d6e-43ee-a612-857a76f8c08a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:05.064307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:05.064272 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46blr\" (UniqueName: \"kubernetes.io/projected/42cec7ba-2d6e-43ee-a612-857a76f8c08a-kube-api-access-46blr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:05.064307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:05.064305 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42cec7ba-2d6e-43ee-a612-857a76f8c08a-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:05.103871 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:26:05.103830 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:26:05.799250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:05.799218 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" Oct 30 19:26:05.799765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:05.799247 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p" event={"ID":"42cec7ba-2d6e-43ee-a612-857a76f8c08a","Type":"ContainerDied","Data":"76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0"} Oct 30 19:26:05.799765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:05.799309 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c59a65df4f42216a1f50ed6889d3dc401ff0af1e360c5325e07eb747177bd0" Oct 30 19:26:06.173871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:06.173722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:07.836856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:07.836736 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.441995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:09.441928 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:11.326297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:11.326250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:12.991136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:12.990548 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:14.036138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.036082 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts"] Oct 30 19:26:14.036697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.036683 4990 cpu_manager.go:414] "containerMap: removing container" podUID="42cec7ba-2d6e-43ee-a612-857a76f8c08a" containerName="revoke-keys" Oct 30 19:26:14.036741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.036698 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cec7ba-2d6e-43ee-a612-857a76f8c08a" containerName="revoke-keys" Oct 30 19:26:14.036805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.036796 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cec7ba-2d6e-43ee-a612-857a76f8c08a" containerName="revoke-keys" Oct 30 19:26:14.039649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.039632 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.057628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.057603 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts"] Oct 30 19:26:14.062545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.062521 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:14.062692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.062581 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:14.186100 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.186064 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.186285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.186123 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbtr\" (UniqueName: \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.186371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.186352 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.288286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.288200 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.288440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.288343 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.288440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.288377 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbtr\" (UniqueName: \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.288940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.288917 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.290736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.290714 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.305361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.305329 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbtr\" (UniqueName: \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\") pod \"maas-api-revoke-sub-1d94e164-tc9ts\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.353438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.353400 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:14.654212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.654189 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts"] Oct 30 19:26:14.656244 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:14.656213 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872f7286_a78a_4cda_8ff9_45a17df32e95.slice/crio-cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9 WatchSource:0}: Error finding container cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9: Status 404 returned error can't find the container with id cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9 Oct 30 19:26:14.806603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.803017 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:14.876085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:14.876017 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" event={"ID":"872f7286-a78a-4cda-8ff9-45a17df32e95","Type":"ContainerStarted","Data":"cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9"} Oct 30 19:26:15.608510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.608475 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:26:15.612619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.612599 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.640571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.640535 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:26:15.708478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708448 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708484 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708591 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708689 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84mq\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708714 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708836 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708882 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.708995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.708980 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.709030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.709005 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.810779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.810740 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.810980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.810793 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.810980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.810869 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.810980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.810898 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.810980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.810960 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811056 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84mq\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811085 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811209 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811247 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811314 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811600 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811622 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.811869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.811849 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.812155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.812132 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.813734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.813704 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.813936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.813917 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.828103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.828067 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84mq\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.828428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.828407 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:15.886966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.886871 4990 generic.go:356] "Generic (PLEG): container finished" podID="872f7286-a78a-4cda-8ff9-45a17df32e95" containerID="6f9477fa35ff5dccbc4a2c4dfc3bd265f06600be83d3f7fbbbd4646c88cea607" exitCode=0 Oct 30 19:26:15.886966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.886919 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" event={"ID":"872f7286-a78a-4cda-8ff9-45a17df32e95","Type":"ContainerDied","Data":"6f9477fa35ff5dccbc4a2c4dfc3bd265f06600be83d3f7fbbbd4646c88cea607"} Oct 30 19:26:15.927368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:15.927336 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:16.263259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.263236 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:26:16.265478 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:16.265450 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7652dfa_4468_4eeb_ac34_b160116cae61.slice/crio-e5e35f43044fed4edcc2049a9af439817cd06724e2e5d83f88077937f30fd0dc WatchSource:0}: Error finding container e5e35f43044fed4edcc2049a9af439817cd06724e2e5d83f88077937f30fd0dc: Status 404 returned error can't find the container with id e5e35f43044fed4edcc2049a9af439817cd06724e2e5d83f88077937f30fd0dc Oct 30 19:26:16.268077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.268046 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:16.268184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.268140 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:16.268184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.268173 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:16.506988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.506456 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:16.905352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.905324 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" event={"ID":"e7652dfa-4468-4eeb-ac34-b160116cae61","Type":"ContainerStarted","Data":"aeac255e7e05459a6f1eb16b0c1f7b6dad083d2ff7399a8691afbe2d4549d68a"} Oct 30 19:26:16.905797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.905780 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" event={"ID":"e7652dfa-4468-4eeb-ac34-b160116cae61","Type":"ContainerStarted","Data":"e5e35f43044fed4edcc2049a9af439817cd06724e2e5d83f88077937f30fd0dc"} Oct 30 19:26:16.927551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.927485 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:16.952815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:16.952764 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podStartSLOduration=1.952751241 podStartE2EDuration="1.952751241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:16.952253067 +0000 UTC m=+1284.663778437" watchObservedRunningTime="2026-10-30 19:26:16.952751241 +0000 UTC m=+1284.664276610" Oct 30 19:26:17.149927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.149901 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:17.342460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.342425 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\" (UniqueName: \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\") pod \"872f7286-a78a-4cda-8ff9-45a17df32e95\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " Oct 30 19:26:17.342704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.342683 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\") pod \"872f7286-a78a-4cda-8ff9-45a17df32e95\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " Oct 30 19:26:17.342778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.342762 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\") pod \"872f7286-a78a-4cda-8ff9-45a17df32e95\" (UID: \"872f7286-a78a-4cda-8ff9-45a17df32e95\") " Oct 30 19:26:17.343132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.343101 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca" pod "872f7286-a78a-4cda-8ff9-45a17df32e95" (UID: "872f7286-a78a-4cda-8ff9-45a17df32e95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:17.343320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.343301 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:17.344765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.344739 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca" pod "872f7286-a78a-4cda-8ff9-45a17df32e95" (UID: "872f7286-a78a-4cda-8ff9-45a17df32e95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:17.344846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.344776 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr" pod "872f7286-a78a-4cda-8ff9-45a17df32e95" (UID: "872f7286-a78a-4cda-8ff9-45a17df32e95"). InnerVolumeSpecName "kube-api-access-vdbtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:17.444282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.444248 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbtr\" (UniqueName: \"kubernetes.io/projected/872f7286-a78a-4cda-8ff9-45a17df32e95-kube-api-access-vdbtr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:17.444499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.444474 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872f7286-a78a-4cda-8ff9-45a17df32e95-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:17.915199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.915169 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" Oct 30 19:26:17.915719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.915185 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts" event={"ID":"872f7286-a78a-4cda-8ff9-45a17df32e95","Type":"ContainerDied","Data":"cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9"} Oct 30 19:26:17.915836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.915749 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf24ddd6a423d24c7a285b2dc1f89ee4ab4f51d1aae13b0726de2d1600ab66f9" Oct 30 19:26:17.929487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.929462 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:17.929607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:17.929507 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:18.189906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:18.189750 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:18.927983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:18.927956 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:18.928408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:18.928004 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:19.928435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:19.928403 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:19.928894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:19.928454 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:20.008396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:20.008359 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.927817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:20.927782 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:20.928019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:20.927833 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:21.737656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:21.737620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:21.928428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:21.928393 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:21.928636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:21.928437 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:22.927735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:22.927703 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:26:22.928253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:22.927759 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 19:26:23.436899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:23.436854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:23.991953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:23.991916 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:23.994707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:23.994687 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:23.996163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:23.996058 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:26:24.100113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.100060 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7"] Oct 30 19:26:24.101895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.101779 4990 cpu_manager.go:414] "containerMap: removing container" podUID="872f7286-a78a-4cda-8ff9-45a17df32e95" containerName="revoke-keys" Oct 30 19:26:24.102081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.102063 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872f7286-a78a-4cda-8ff9-45a17df32e95" containerName="revoke-keys" Oct 30 19:26:24.102522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.102504 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="872f7286-a78a-4cda-8ff9-45a17df32e95" containerName="revoke-keys" Oct 30 19:26:24.114919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.114891 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.177876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.177826 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:24.178913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.178891 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:24.220312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.220282 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7"] Oct 30 19:26:24.249869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.249785 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.250209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.250185 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb44l\" (UniqueName: \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.250470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.250441 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.351888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.351853 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.352132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.351959 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb44l\" (UniqueName: \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.352132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.352059 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.352745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.352724 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.354993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.354960 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.369518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.369492 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb44l\" (UniqueName: \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\") pod \"maas-api-revoke-sub-2a30ea50-9jkr7\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:24.562204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:24.557318 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:25.625123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.620489 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7"] Oct 30 19:26:25.688681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.683953 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:26:25.735281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.732121 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.774782 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.774868 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.774894 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.774996 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.775040 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.797118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.775122 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjq4\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.837117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.805218 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.837117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.805298 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.837117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.805347 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.866233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.866182 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:25.896577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.896537 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:26:25.928121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.928008 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.928121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.928060 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.952117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.947775 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.972912 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.972988 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973069 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjq4\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973194 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973246 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973288 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973336 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.973766 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:25.976131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:25.974818 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.037690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.037603 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:26.038108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.038058 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjq4\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.038625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.038596 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.039047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.038986 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.192294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.190627 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.192294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.191853 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.338114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.333175 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" event={"ID":"22ed9832-f818-49c0-b93a-f878c86421e5","Type":"ContainerStarted","Data":"9f48d84a7059860070cc025f32a1368488156753ca2044b00cda0f9cc3d2dc7e"} Oct 30 19:26:26.482118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.481434 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.656114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.653199 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:26.681116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.678395 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:26.866538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.866492 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:26.877176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:26.869002 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:27.033761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.033724 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:27.099282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.099234 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:27.343185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.342739 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:26:27.432517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.432473 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" event={"ID":"22ed9832-f818-49c0-b93a-f878c86421e5","Type":"ContainerStarted","Data":"5ce6947aafdf1c0a776db9f17b7db7378007ef878b9bea32426a1c981385bfe9"} Oct 30 19:26:27.454449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.453343 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:27.487634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.487568 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" podStartSLOduration=3.487549749 podStartE2EDuration="3.487549749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:27.483244976 +0000 UTC m=+1295.194770362" watchObservedRunningTime="2026-10-30 19:26:27.487549749 +0000 UTC m=+1295.199075116" Oct 30 19:26:27.553041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.553013 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:26:27.558912 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:27.558871 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce1795b_beee_4956_99f2_dc7fbbb89954.slice/crio-25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf WatchSource:0}: Error finding container 25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf: Status 404 returned error can't find the container with id 25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf Oct 30 19:26:27.562165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.562121 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:27.562313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.562212 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:27.562313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:27.562260 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:26:28.441184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.441150 4990 generic.go:356] "Generic (PLEG): container finished" podID="22ed9832-f818-49c0-b93a-f878c86421e5" containerID="5ce6947aafdf1c0a776db9f17b7db7378007ef878b9bea32426a1c981385bfe9" exitCode=0 Oct 30 19:26:28.441606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.441234 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" event={"ID":"22ed9832-f818-49c0-b93a-f878c86421e5","Type":"ContainerDied","Data":"5ce6947aafdf1c0a776db9f17b7db7378007ef878b9bea32426a1c981385bfe9"} Oct 30 19:26:28.442995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.442964 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" event={"ID":"3ce1795b-beee-4956-99f2-dc7fbbb89954","Type":"ContainerStarted","Data":"ecf78cbe34512b37d08dbf8cf4cad53f62122ef2b3b9fadd72a4967721b1efcb"} Oct 30 19:26:28.443187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.443014 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" event={"ID":"3ce1795b-beee-4956-99f2-dc7fbbb89954","Type":"ContainerStarted","Data":"25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf"} Oct 30 19:26:28.521157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.521073 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podStartSLOduration=3.521055923 podStartE2EDuration="3.521055923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:28.519880851 +0000 UTC m=+1296.231406237" watchObservedRunningTime="2026-10-30 19:26:28.521055923 +0000 UTC m=+1296.232581290" Oct 30 19:26:28.688556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.688518 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:28.693288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.693204 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:28.693549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:28.693521 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:29.124234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.124198 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:29.697175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.696428 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:29.697175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.696515 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:29.702680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.702610 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:29.829027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.828988 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\" (UniqueName: \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\") pod \"22ed9832-f818-49c0-b93a-f878c86421e5\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " Oct 30 19:26:29.829235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.829147 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\") pod \"22ed9832-f818-49c0-b93a-f878c86421e5\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " Oct 30 19:26:29.829338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.829318 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\") pod \"22ed9832-f818-49c0-b93a-f878c86421e5\" (UID: \"22ed9832-f818-49c0-b93a-f878c86421e5\") " Oct 30 19:26:29.829799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.829777 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca" pod "22ed9832-f818-49c0-b93a-f878c86421e5" (UID: "22ed9832-f818-49c0-b93a-f878c86421e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:29.830006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.829991 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:29.831562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.831537 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca" pod "22ed9832-f818-49c0-b93a-f878c86421e5" (UID: "22ed9832-f818-49c0-b93a-f878c86421e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:29.831696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.831579 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l" pod "22ed9832-f818-49c0-b93a-f878c86421e5" (UID: "22ed9832-f818-49c0-b93a-f878c86421e5"). InnerVolumeSpecName "kube-api-access-fb44l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:29.931506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.931459 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb44l\" (UniqueName: \"kubernetes.io/projected/22ed9832-f818-49c0-b93a-f878c86421e5-kube-api-access-fb44l\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:29.931506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:29.931489 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ed9832-f818-49c0-b93a-f878c86421e5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:30.460144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:30.460116 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" Oct 30 19:26:30.460144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:30.460125 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7" event={"ID":"22ed9832-f818-49c0-b93a-f878c86421e5","Type":"ContainerDied","Data":"9f48d84a7059860070cc025f32a1368488156753ca2044b00cda0f9cc3d2dc7e"} Oct 30 19:26:30.460349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:30.460214 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f48d84a7059860070cc025f32a1368488156753ca2044b00cda0f9cc3d2dc7e" Oct 30 19:26:30.680171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:30.679446 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:30.680171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:30.679509 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:31.026958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:31.026850 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:31.686117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:31.683462 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:31.686117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:31.683535 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:32.679615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:32.679578 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:32.680254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:32.680226 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:33.099347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:33.099310 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.687302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:33.682437 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:33.687302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:33.682509 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:34.679421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:34.679381 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:26:34.679629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:34.679452 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:26:35.051574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:35.051539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:35.110974 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:26:35.110933 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:26:35.684417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:35.684388 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:35.684877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:35.684855 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:35.685508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:35.685491 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:26:36.844322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.844283 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:36.988494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.988458 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h"] Oct 30 19:26:36.989054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.989041 4990 cpu_manager.go:414] "containerMap: removing container" podUID="22ed9832-f818-49c0-b93a-f878c86421e5" containerName="revoke-keys" Oct 30 19:26:36.989114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.989055 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ed9832-f818-49c0-b93a-f878c86421e5" containerName="revoke-keys" Oct 30 19:26:36.989182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.989173 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ed9832-f818-49c0-b93a-f878c86421e5" containerName="revoke-keys" Oct 30 19:26:36.993930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:36.993911 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.012913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.012888 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h"] Oct 30 19:26:37.017253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.017227 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:37.017600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.017241 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:37.155305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.155215 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.155305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.155276 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.155520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.155354 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl54d\" (UniqueName: \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.256254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.256219 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.256430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.256302 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.256430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.256332 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl54d\" (UniqueName: \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.257008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.256982 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.258969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.258945 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.274758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.274727 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl54d\" (UniqueName: \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\") pod \"maas-api-revoke-sub-fda86ab9-q4f2h\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.307212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.307185 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:37.616845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:37.616807 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h"] Oct 30 19:26:37.617392 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:37.617364 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0575d3bf_f0a3_4673_94b4_787a8d7c282f.slice/crio-1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a WatchSource:0}: Error finding container 1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a: Status 404 returned error can't find the container with id 1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a Oct 30 19:26:38.535158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:38.534949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:38.554997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:38.554535 4990 generic.go:356] "Generic (PLEG): container finished" podID="0575d3bf-f0a3-4673-94b4-787a8d7c282f" containerID="fb62533cbd513b3311d65f20d1294da3df37a908626519af1451fc69d202d63e" exitCode=0 Oct 30 19:26:38.554997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:38.554620 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" event={"ID":"0575d3bf-f0a3-4673-94b4-787a8d7c282f","Type":"ContainerDied","Data":"fb62533cbd513b3311d65f20d1294da3df37a908626519af1451fc69d202d63e"} Oct 30 19:26:38.554997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:38.554682 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" event={"ID":"0575d3bf-f0a3-4673-94b4-787a8d7c282f","Type":"ContainerStarted","Data":"1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a"} Oct 30 19:26:39.732936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.732909 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:39.789539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.789503 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\" (UniqueName: \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\") pod \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " Oct 30 19:26:39.789716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.789588 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\") pod \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " Oct 30 19:26:39.789809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.789757 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\") pod \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\" (UID: \"0575d3bf-f0a3-4673-94b4-787a8d7c282f\") " Oct 30 19:26:39.790418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.790381 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca" pod "0575d3bf-f0a3-4673-94b4-787a8d7c282f" (UID: "0575d3bf-f0a3-4673-94b4-787a8d7c282f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:39.792572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.792541 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca" pod "0575d3bf-f0a3-4673-94b4-787a8d7c282f" (UID: "0575d3bf-f0a3-4673-94b4-787a8d7c282f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:39.792701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.792671 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d" pod "0575d3bf-f0a3-4673-94b4-787a8d7c282f" (UID: "0575d3bf-f0a3-4673-94b4-787a8d7c282f"). InnerVolumeSpecName "kube-api-access-sl54d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:39.893040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.892941 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl54d\" (UniqueName: \"kubernetes.io/projected/0575d3bf-f0a3-4673-94b4-787a8d7c282f-kube-api-access-sl54d\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:39.893040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.892987 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:39.893040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:39.893003 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0575d3bf-f0a3-4673-94b4-787a8d7c282f-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:40.213902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:40.213601 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:40.573757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:40.573727 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" Oct 30 19:26:40.573937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:40.573760 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h" event={"ID":"0575d3bf-f0a3-4673-94b4-787a8d7c282f","Type":"ContainerDied","Data":"1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a"} Oct 30 19:26:40.573937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:40.573808 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7865b961fdbc66e6d53f4b9c3aed1db3b7325d1c5cc5091ff72f3d16efef1a" Oct 30 19:26:42.047703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:42.047661 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.488116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.463049 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx"] Oct 30 19:26:44.562118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.533546 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0575d3bf-f0a3-4673-94b4-787a8d7c282f" containerName="revoke-keys" Oct 30 19:26:44.562118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.533577 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0575d3bf-f0a3-4673-94b4-787a8d7c282f" containerName="revoke-keys" Oct 30 19:26:44.562118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.533744 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0575d3bf-f0a3-4673-94b4-787a8d7c282f" containerName="revoke-keys" Oct 30 19:26:44.733116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.722393 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.810222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.774427 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:44.810222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.774580 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:44.810222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.777224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.818362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.810775 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.818362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.810856 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.818362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.811065 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7gq\" (UniqueName: \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.925233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.924750 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx"] Oct 30 19:26:44.941116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.928030 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.941116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.928124 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.941116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.928322 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7gq\" (UniqueName: \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.941116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.929381 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.966501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.966467 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7gq\" (UniqueName: \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:44.967528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:44.967459 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fcf7fd3-9qkqx\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:45.121297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.121189 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.231347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.231305 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:45.305277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.305223 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.539307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.538606 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:26:45.540202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.539851 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.545186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.545154 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:26:45.545655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.545347 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:26:45.643889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.643173 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.684805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.684168 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.684805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:45.684269 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:26:46.124112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.114189 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.309303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.124986 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.310560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.157206 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.317203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.157271 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 19:26:46.341138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.317431 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:26:46.341138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.306843 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.341138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.323629 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:26:46.543962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.543836 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.591250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.553183 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:26:46.591250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.556359 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.591250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.556621 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.662133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.643550 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:26:46.751876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.709033 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.811177 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:46.807252 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b6da5_3216_4ceb_9e67_f87202ad6dbd.slice/crio-8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf WatchSource:0}: Error finding container 8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf: Status 404 returned error can't find the container with id 8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf Oct 30 19:26:46.826063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.826030 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:26:46.826659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.826638 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:46.826827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.826813 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:26:46.826971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.826958 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:26:46.827114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.827083 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:26:46.827241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.827229 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:26:46.851356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:46.848167 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:47.226206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.220220 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx"] Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.307272 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.318143 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.319674 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.328627 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.328946 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:26:47.371116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.331232 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:47.584366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.553681 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:26:47.584366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.558274 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:47.584366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.558421 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:26:47.584366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.558597 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:26:47.720982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.720946 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:26:47.735398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.735346 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:26:47.899154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:47.899047 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" event={"ID":"430b6da5-3216-4ceb-9e67-f87202ad6dbd","Type":"ContainerStarted","Data":"8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf"} Oct 30 19:26:48.053111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.038873 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:26:48.053111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.039126 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:26:48.086158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.085432 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:26:48.110739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.110713 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:26:48.113690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.113665 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:26:48.115744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.115723 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:26:48.252109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.229651 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:26:48.252109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.247310 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:26:48.252109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.247359 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:26:48.252109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.247398 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:26:48.910567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.910533 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" event={"ID":"430b6da5-3216-4ceb-9e67-f87202ad6dbd","Type":"ContainerStarted","Data":"7172f8867aa6b41c4222c3032d0c56560d5abddf8d2f64e2da8f418bda3b1955"} Oct 30 19:26:48.946815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:48.946758 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" podStartSLOduration=4.9467394129999995 podStartE2EDuration="4.946739413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:48.945068701 +0000 UTC m=+1316.656594067" watchObservedRunningTime="2026-10-30 19:26:48.946739413 +0000 UTC m=+1316.658264779" Oct 30 19:26:49.920608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:49.920569 4990 generic.go:356] "Generic (PLEG): container finished" podID="430b6da5-3216-4ceb-9e67-f87202ad6dbd" containerID="7172f8867aa6b41c4222c3032d0c56560d5abddf8d2f64e2da8f418bda3b1955" exitCode=0 Oct 30 19:26:49.921007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:49.920652 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" event={"ID":"430b6da5-3216-4ceb-9e67-f87202ad6dbd","Type":"ContainerDied","Data":"7172f8867aa6b41c4222c3032d0c56560d5abddf8d2f64e2da8f418bda3b1955"} Oct 30 19:26:49.943837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:49.943786 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:51.087899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.087868 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:51.196016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.195910 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\" (UniqueName: \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\") pod \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " Oct 30 19:26:51.196221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.196084 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\") pod \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " Oct 30 19:26:51.196293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.196252 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\") pod \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\" (UID: \"430b6da5-3216-4ceb-9e67-f87202ad6dbd\") " Oct 30 19:26:51.196643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.196617 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca" pod "430b6da5-3216-4ceb-9e67-f87202ad6dbd" (UID: "430b6da5-3216-4ceb-9e67-f87202ad6dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:51.196917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.196896 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:51.198594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.198569 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca" pod "430b6da5-3216-4ceb-9e67-f87202ad6dbd" (UID: "430b6da5-3216-4ceb-9e67-f87202ad6dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:51.198723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.198605 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq" pod "430b6da5-3216-4ceb-9e67-f87202ad6dbd" (UID: "430b6da5-3216-4ceb-9e67-f87202ad6dbd"). InnerVolumeSpecName "kube-api-access-jl7gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:51.298538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.298503 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/430b6da5-3216-4ceb-9e67-f87202ad6dbd-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:51.298538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.298537 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7gq\" (UniqueName: \"kubernetes.io/projected/430b6da5-3216-4ceb-9e67-f87202ad6dbd-kube-api-access-jl7gq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:51.589424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.589389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:51.938447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.938370 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" Oct 30 19:26:51.938447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.938408 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx" event={"ID":"430b6da5-3216-4ceb-9e67-f87202ad6dbd","Type":"ContainerDied","Data":"8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf"} Oct 30 19:26:51.938629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.938456 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d49f2ed207297f39e1be798ecf94c82a7112153718bef6611fb928255da0cdf" Oct 30 19:26:51.974141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.974111 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84"] Oct 30 19:26:51.974787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.974764 4990 cpu_manager.go:414] "containerMap: removing container" podUID="430b6da5-3216-4ceb-9e67-f87202ad6dbd" containerName="revoke-keys" Oct 30 19:26:51.974902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.974787 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430b6da5-3216-4ceb-9e67-f87202ad6dbd" containerName="revoke-keys" Oct 30 19:26:51.974962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.974923 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="430b6da5-3216-4ceb-9e67-f87202ad6dbd" containerName="revoke-keys" Oct 30 19:26:51.978480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:51.978461 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.002166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.002142 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:52.002280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.002243 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:52.009728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.009699 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.009914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.009895 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.010020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.009996 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5l6c\" (UniqueName: \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.016607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.016580 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84"] Oct 30 19:26:52.111611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.111574 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.112137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.111762 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.112137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.111821 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5l6c\" (UniqueName: \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.112411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.112389 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.114569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.114543 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.132593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.132559 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5l6c\" (UniqueName: \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\") pod \"maas-api-revoke-sub-958634c2-r2n84\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.292320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.292226 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:52.595116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.595070 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84"] Oct 30 19:26:52.596392 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:52.596362 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc0d220_4650_40d9_9342_2fe875bd2e51.slice/crio-d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f WatchSource:0}: Error finding container d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f: Status 404 returned error can't find the container with id d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f Oct 30 19:26:52.946069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.946029 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" event={"ID":"7fc0d220-4650-40d9-9342-2fe875bd2e51","Type":"ContainerStarted","Data":"cb9d808a1ad844d2b7f641c6e7211fec23eb8e858e1fc1aa4f4b5a3e6f74dc4d"} Oct 30 19:26:52.946236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.946086 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" event={"ID":"7fc0d220-4650-40d9-9342-2fe875bd2e51","Type":"ContainerStarted","Data":"d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f"} Oct 30 19:26:52.981965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:52.981907 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" podStartSLOduration=1.981888236 podStartE2EDuration="1.981888236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:52.978901724 +0000 UTC m=+1320.690427102" watchObservedRunningTime="2026-10-30 19:26:52.981888236 +0000 UTC m=+1320.693413603" Oct 30 19:26:53.345682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:53.341171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:53.957203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:53.957171 4990 generic.go:356] "Generic (PLEG): container finished" podID="7fc0d220-4650-40d9-9342-2fe875bd2e51" containerID="cb9d808a1ad844d2b7f641c6e7211fec23eb8e858e1fc1aa4f4b5a3e6f74dc4d" exitCode=0 Oct 30 19:26:53.957497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:53.957260 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" event={"ID":"7fc0d220-4650-40d9-9342-2fe875bd2e51","Type":"ContainerDied","Data":"cb9d808a1ad844d2b7f641c6e7211fec23eb8e858e1fc1aa4f4b5a3e6f74dc4d"} Oct 30 19:26:55.146182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.145180 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:55.160124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.160084 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:55.253039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.252959 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\") pod \"7fc0d220-4650-40d9-9342-2fe875bd2e51\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " Oct 30 19:26:55.253039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.253027 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\") pod \"7fc0d220-4650-40d9-9342-2fe875bd2e51\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " Oct 30 19:26:55.253288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.253127 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\" (UniqueName: \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\") pod \"7fc0d220-4650-40d9-9342-2fe875bd2e51\" (UID: \"7fc0d220-4650-40d9-9342-2fe875bd2e51\") " Oct 30 19:26:55.253457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.253427 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca" pod "7fc0d220-4650-40d9-9342-2fe875bd2e51" (UID: "7fc0d220-4650-40d9-9342-2fe875bd2e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:55.253857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.253830 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:55.255280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.255252 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca" pod "7fc0d220-4650-40d9-9342-2fe875bd2e51" (UID: "7fc0d220-4650-40d9-9342-2fe875bd2e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:55.255373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.255259 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c" pod "7fc0d220-4650-40d9-9342-2fe875bd2e51" (UID: "7fc0d220-4650-40d9-9342-2fe875bd2e51"). InnerVolumeSpecName "kube-api-access-n5l6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:55.355030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.354982 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc0d220-4650-40d9-9342-2fe875bd2e51-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:55.355030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.355027 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5l6c\" (UniqueName: \"kubernetes.io/projected/7fc0d220-4650-40d9-9342-2fe875bd2e51-kube-api-access-n5l6c\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:26:55.978481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.978451 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" Oct 30 19:26:55.978668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.978488 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84" event={"ID":"7fc0d220-4650-40d9-9342-2fe875bd2e51","Type":"ContainerDied","Data":"d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f"} Oct 30 19:26:55.978668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:55.978550 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f4be3afe131d453c5145ac52fe60ff3aa005200c50ae752729ef4bbe6f9b4f" Oct 30 19:26:56.780766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:56.780725 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.273766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.273731 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld"] Oct 30 19:26:57.274443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.274424 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7fc0d220-4650-40d9-9342-2fe875bd2e51" containerName="revoke-keys" Oct 30 19:26:57.274508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.274444 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc0d220-4650-40d9-9342-2fe875bd2e51" containerName="revoke-keys" Oct 30 19:26:57.274577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.274567 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc0d220-4650-40d9-9342-2fe875bd2e51" containerName="revoke-keys" Oct 30 19:26:57.278368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.278349 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.300970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.300932 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:26:57.300970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.300931 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:57.308333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.308294 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld"] Oct 30 19:26:57.382459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.382423 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.382637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.382475 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6gg\" (UniqueName: \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.382637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.382589 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.484070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.484034 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6gg\" (UniqueName: \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.484246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.484140 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.484354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.484337 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.484862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.484837 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.486719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.486694 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.500611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.500583 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6gg\" (UniqueName: \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\") pod \"maas-api-revoke-sub-70fb3b09-64nld\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.592534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.592499 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:26:57.908126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.904359 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld"] Oct 30 19:26:57.908126 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:26:57.904450 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ed2791_9194_4ed7_a4bb_e10434384b7d.slice/crio-2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35 WatchSource:0}: Error finding container 2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35: Status 404 returned error can't find the container with id 2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35 Oct 30 19:26:57.995684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:57.995648 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" event={"ID":"d5ed2791-9194-4ed7-a4bb-e10434384b7d","Type":"ContainerStarted","Data":"2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35"} Oct 30 19:26:58.405405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:58.405366 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:58.554023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:58.553988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:59.004604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:59.004567 4990 generic.go:356] "Generic (PLEG): container finished" podID="d5ed2791-9194-4ed7-a4bb-e10434384b7d" containerID="6d561427d7e396bcd5c331a1dedd4f21aa58923235d211b13b4d4e1c41b80a79" exitCode=0 Oct 30 19:26:59.005029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:26:59.004663 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" event={"ID":"d5ed2791-9194-4ed7-a4bb-e10434384b7d","Type":"ContainerDied","Data":"6d561427d7e396bcd5c331a1dedd4f21aa58923235d211b13b4d4e1c41b80a79"} Oct 30 19:27:00.113371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.113063 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:00.195246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.195221 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:27:00.330895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.330860 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\" (UniqueName: \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\") pod \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " Oct 30 19:27:00.331138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.330998 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\") pod \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " Oct 30 19:27:00.331224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.331197 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\") pod \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\" (UID: \"d5ed2791-9194-4ed7-a4bb-e10434384b7d\") " Oct 30 19:27:00.331510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.331484 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca" pod "d5ed2791-9194-4ed7-a4bb-e10434384b7d" (UID: "d5ed2791-9194-4ed7-a4bb-e10434384b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:00.331650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.331628 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:00.333618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.333590 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca" pod "d5ed2791-9194-4ed7-a4bb-e10434384b7d" (UID: "d5ed2791-9194-4ed7-a4bb-e10434384b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:00.333736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.333602 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg" pod "d5ed2791-9194-4ed7-a4bb-e10434384b7d" (UID: "d5ed2791-9194-4ed7-a4bb-e10434384b7d"). InnerVolumeSpecName "kube-api-access-tn6gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:00.433137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.433083 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5ed2791-9194-4ed7-a4bb-e10434384b7d-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:00.433137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:00.433133 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6gg\" (UniqueName: \"kubernetes.io/projected/d5ed2791-9194-4ed7-a4bb-e10434384b7d-kube-api-access-tn6gg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:01.021166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:01.021132 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" event={"ID":"d5ed2791-9194-4ed7-a4bb-e10434384b7d","Type":"ContainerDied","Data":"2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35"} Oct 30 19:27:01.021318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:01.021189 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a16e70a1d57bca01b28c04bb0992a41ef77bf665fd7b73e7a5af6116245ed35" Oct 30 19:27:01.021318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:01.021221 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld" Oct 30 19:27:01.718814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:01.718782 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:01.869402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:01.869368 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:02.639907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:02.639870 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:02.801809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:02.801770 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.584751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:03.584711 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:04.005829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.005747 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h"] Oct 30 19:27:04.006355 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.006340 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d5ed2791-9194-4ed7-a4bb-e10434384b7d" containerName="revoke-keys" Oct 30 19:27:04.006399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.006356 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ed2791-9194-4ed7-a4bb-e10434384b7d" containerName="revoke-keys" Oct 30 19:27:04.006461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.006452 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ed2791-9194-4ed7-a4bb-e10434384b7d" containerName="revoke-keys" Oct 30 19:27:04.010198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.010178 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.027247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.027224 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:04.037320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.037286 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:04.046143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.044500 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h"] Oct 30 19:27:04.182803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.182761 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72ps\" (UniqueName: \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.182990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.182876 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.183151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.183086 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.284287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.284191 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72ps\" (UniqueName: \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.284287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.284250 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.284533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.284367 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.284945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.284919 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.286700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.286669 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.302671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.302641 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72ps\" (UniqueName: \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\") pod \"maas-api-revoke-sub-4a586460-dp64h\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.324450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.324415 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:04.647265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:04.647239 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h"] Oct 30 19:27:04.648307 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:04.648277 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58d5150_961e_4b0d_9468_adc3633c8e36.slice/crio-b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355 WatchSource:0}: Error finding container b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355: Status 404 returned error can't find the container with id b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355 Oct 30 19:27:05.106114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:05.106058 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" event={"ID":"b58d5150-961e-4b0d-9468-adc3633c8e36","Type":"ContainerStarted","Data":"b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355"} Oct 30 19:27:05.183151 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:27:05.176289 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:27:05.763128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:05.762790 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.162116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.161960 4990 generic.go:356] "Generic (PLEG): container finished" podID="b58d5150-961e-4b0d-9468-adc3633c8e36" containerID="2f1b3d15b9d053ad5dbb82407024d6477293f6507c9fce8985377e73872c6955" exitCode=0 Oct 30 19:27:06.172112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.168505 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" event={"ID":"b58d5150-961e-4b0d-9468-adc3633c8e36","Type":"ContainerDied","Data":"2f1b3d15b9d053ad5dbb82407024d6477293f6507c9fce8985377e73872c6955"} Oct 30 19:27:06.563112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.560489 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:06.598170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.597701 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:06.602398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.602365 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:06.661237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.661191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:06.731115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.730263 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:06.831175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:06.825357 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.046116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.036202 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.186549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.186447 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:27:07.239209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.239165 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:07.335225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.335177 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.597762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.595192 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.597762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.597187 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:07.632518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.632477 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.697178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.697134 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.773118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.749747 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:07.773118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.749818 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:07.773118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:07.749853 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:08.062706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.062676 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:08.328214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.316052 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:08.366414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.366385 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:08.522964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.522928 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\" (UniqueName: \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\") pod \"b58d5150-961e-4b0d-9468-adc3633c8e36\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " Oct 30 19:27:08.523191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.523014 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\") pod \"b58d5150-961e-4b0d-9468-adc3633c8e36\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " Oct 30 19:27:08.523281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.523216 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\") pod \"b58d5150-961e-4b0d-9468-adc3633c8e36\" (UID: \"b58d5150-961e-4b0d-9468-adc3633c8e36\") " Oct 30 19:27:08.523450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.523424 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca" pod "b58d5150-961e-4b0d-9468-adc3633c8e36" (UID: "b58d5150-961e-4b0d-9468-adc3633c8e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:08.524229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.524200 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:08.525086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.525065 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps" pod "b58d5150-961e-4b0d-9468-adc3633c8e36" (UID: "b58d5150-961e-4b0d-9468-adc3633c8e36"). InnerVolumeSpecName "kube-api-access-h72ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:08.525222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.525206 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca" pod "b58d5150-961e-4b0d-9468-adc3633c8e36" (UID: "b58d5150-961e-4b0d-9468-adc3633c8e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:08.626002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.625912 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b58d5150-961e-4b0d-9468-adc3633c8e36-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:08.626002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:08.625945 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72ps\" (UniqueName: \"kubernetes.io/projected/b58d5150-961e-4b0d-9468-adc3633c8e36-kube-api-access-h72ps\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:09.024498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.024406 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:09.149102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.149050 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h"] Oct 30 19:27:09.150004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.149982 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b58d5150-961e-4b0d-9468-adc3633c8e36" containerName="revoke-keys" Oct 30 19:27:09.150072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.150008 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58d5150-961e-4b0d-9468-adc3633c8e36" containerName="revoke-keys" Oct 30 19:27:09.150208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.150193 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58d5150-961e-4b0d-9468-adc3633c8e36" containerName="revoke-keys" Oct 30 19:27:09.154187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.154162 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.177421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.173361 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h"] Oct 30 19:27:09.335591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.335558 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.336019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.335622 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.336019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.335726 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tqv\" (UniqueName: \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.356039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.356011 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" Oct 30 19:27:09.356218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.356043 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h" event={"ID":"b58d5150-961e-4b0d-9468-adc3633c8e36","Type":"ContainerDied","Data":"b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355"} Oct 30 19:27:09.356218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.356141 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a595144ea7287f8fdcb1c02d0974240bcc337a5c018e53ef1e04a6b7b22355" Oct 30 19:27:09.437327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.437292 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.437492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.437376 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.437492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.437406 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tqv\" (UniqueName: \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.438024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.437996 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.439749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.439727 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.455045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.455017 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tqv\" (UniqueName: \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\") pod \"maas-api-revoke-sub-89ede2e7-tbq5h\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.472134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.472111 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:09.779974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:09.779945 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h"] Oct 30 19:27:09.782580 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:09.782546 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5b2016_4b60_4abb_bd42_633f299ddaee.slice/crio-40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b WatchSource:0}: Error finding container 40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b: Status 404 returned error can't find the container with id 40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b Oct 30 19:27:10.364685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:10.364587 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" event={"ID":"3a5b2016-4b60-4abb-bd42-633f299ddaee","Type":"ContainerStarted","Data":"282d56c2b6921777ac268a9a1463843a67b52abe7002765c1daebc9c62703dfb"} Oct 30 19:27:10.364685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:10.364651 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" event={"ID":"3a5b2016-4b60-4abb-bd42-633f299ddaee","Type":"ContainerStarted","Data":"40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b"} Oct 30 19:27:10.402031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:10.401957 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" podStartSLOduration=1.401938696 podStartE2EDuration="1.401938696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:10.398299397 +0000 UTC m=+1338.109824764" watchObservedRunningTime="2026-10-30 19:27:10.401938696 +0000 UTC m=+1338.113464062" Oct 30 19:27:10.589329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:10.589298 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:11.378626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:11.378590 4990 generic.go:356] "Generic (PLEG): container finished" podID="3a5b2016-4b60-4abb-bd42-633f299ddaee" containerID="282d56c2b6921777ac268a9a1463843a67b52abe7002765c1daebc9c62703dfb" exitCode=0 Oct 30 19:27:11.379171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:11.378643 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" event={"ID":"3a5b2016-4b60-4abb-bd42-633f299ddaee","Type":"ContainerDied","Data":"282d56c2b6921777ac268a9a1463843a67b52abe7002765c1daebc9c62703dfb"} Oct 30 19:27:12.268065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.268027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:12.554457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.554430 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:12.582593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.582563 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\") pod \"3a5b2016-4b60-4abb-bd42-633f299ddaee\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " Oct 30 19:27:12.582731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.582610 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\") pod \"3a5b2016-4b60-4abb-bd42-633f299ddaee\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " Oct 30 19:27:12.582731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.582651 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\" (UniqueName: \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\") pod \"3a5b2016-4b60-4abb-bd42-633f299ddaee\" (UID: \"3a5b2016-4b60-4abb-bd42-633f299ddaee\") " Oct 30 19:27:12.582974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.582945 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca" pod "3a5b2016-4b60-4abb-bd42-633f299ddaee" (UID: "3a5b2016-4b60-4abb-bd42-633f299ddaee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:12.583369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.583333 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:12.584706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.584676 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca" pod "3a5b2016-4b60-4abb-bd42-633f299ddaee" (UID: "3a5b2016-4b60-4abb-bd42-633f299ddaee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:12.584915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.584890 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv" pod "3a5b2016-4b60-4abb-bd42-633f299ddaee" (UID: "3a5b2016-4b60-4abb-bd42-633f299ddaee"). InnerVolumeSpecName "kube-api-access-v8tqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:12.684247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.684215 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5b2016-4b60-4abb-bd42-633f299ddaee-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:12.684247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:12.684242 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tqv\" (UniqueName: \"kubernetes.io/projected/3a5b2016-4b60-4abb-bd42-633f299ddaee-kube-api-access-v8tqv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:13.397950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:13.397923 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" Oct 30 19:27:13.398187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:13.397959 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h" event={"ID":"3a5b2016-4b60-4abb-bd42-633f299ddaee","Type":"ContainerDied","Data":"40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b"} Oct 30 19:27:13.398187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:13.398008 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40db0a015a1c2cbd4cc75c3e2a38b6db8474bc999f3ae1fe9375c67e2070274b" Oct 30 19:27:14.065748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:14.065710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:15.680370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:15.680330 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:16.834009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.833974 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7"] Oct 30 19:27:16.834575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.834559 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3a5b2016-4b60-4abb-bd42-633f299ddaee" containerName="revoke-keys" Oct 30 19:27:16.834632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.834576 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5b2016-4b60-4abb-bd42-633f299ddaee" containerName="revoke-keys" Oct 30 19:27:16.834701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.834691 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5b2016-4b60-4abb-bd42-633f299ddaee" containerName="revoke-keys" Oct 30 19:27:16.839365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.839339 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:16.857402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.857371 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7"] Oct 30 19:27:16.860762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.860739 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:16.860920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.860887 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:16.938627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.938585 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:16.938794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.938770 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:16.938857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:16.938831 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkxm\" (UniqueName: \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.039878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.039846 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.040053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.039932 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkxm\" (UniqueName: \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.040107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.040054 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.040722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.040691 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.042735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.042708 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.059125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.059077 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkxm\" (UniqueName: \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\") pod \"maas-api-revoke-sub-cf20cd35-2dtq7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.163115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.163001 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:17.394831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.394783 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:17.482316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.482280 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7"] Oct 30 19:27:17.957724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.957689 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p"] Oct 30 19:27:17.962014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.961996 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:17.980479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:17.980452 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p"] Oct 30 19:27:18.053945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.053913 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.054119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.053955 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwhd\" (UniqueName: \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.054229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.054205 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.155477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.155379 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.155649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.155552 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.155649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.155592 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwhd\" (UniqueName: \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.156242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.156216 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.157744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.157718 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.172436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.172416 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwhd\" (UniqueName: \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\") pod \"maas-api-revoke-sub-09d2bbc6-t6k9p\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.276627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.276594 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:18.442113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.442009 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" event={"ID":"08b0e10d-41af-454f-8e7c-fcc729e9fda7","Type":"ContainerStarted","Data":"9101a423d241e1f76db9da7e60d7fbe278d5e7682ddfee929a727f69de65d5b8"} Oct 30 19:27:18.442113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.442069 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" event={"ID":"08b0e10d-41af-454f-8e7c-fcc729e9fda7","Type":"ContainerStarted","Data":"cae809724f5aa2792a645b30384fa261b9c0e7ab092691ca2b43c748253d38b9"} Oct 30 19:27:18.478904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.478851 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" podStartSLOduration=2.478836969 podStartE2EDuration="2.478836969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:18.475392761 +0000 UTC m=+1346.186918127" watchObservedRunningTime="2026-10-30 19:27:18.478836969 +0000 UTC m=+1346.190362335" Oct 30 19:27:18.589898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:18.589865 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p"] Oct 30 19:27:18.590235 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:18.590212 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012339dd_35cc_4f80_99d9_c9091d69f6df.slice/crio-e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e WatchSource:0}: Error finding container e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e: Status 404 returned error can't find the container with id e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e Oct 30 19:27:19.091447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.091123 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:19.453790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.453754 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" event={"ID":"012339dd-35cc-4f80-99d9-c9091d69f6df","Type":"ContainerStarted","Data":"c8fcf9f6053105c6a4256ee8d24eda2a475463b984899c59855b705cb6dc9127"} Oct 30 19:27:19.453976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.453814 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" event={"ID":"012339dd-35cc-4f80-99d9-c9091d69f6df","Type":"ContainerStarted","Data":"e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e"} Oct 30 19:27:19.455750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.455722 4990 generic.go:356] "Generic (PLEG): container finished" podID="08b0e10d-41af-454f-8e7c-fcc729e9fda7" containerID="9101a423d241e1f76db9da7e60d7fbe278d5e7682ddfee929a727f69de65d5b8" exitCode=0 Oct 30 19:27:19.455870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.455804 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" event={"ID":"08b0e10d-41af-454f-8e7c-fcc729e9fda7","Type":"ContainerDied","Data":"9101a423d241e1f76db9da7e60d7fbe278d5e7682ddfee929a727f69de65d5b8"} Oct 30 19:27:19.495372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:19.495325 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" podStartSLOduration=2.495311036 podStartE2EDuration="2.495311036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:19.488008951 +0000 UTC m=+1347.199534314" watchObservedRunningTime="2026-10-30 19:27:19.495311036 +0000 UTC m=+1347.206836450" Oct 30 19:27:20.465286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.465253 4990 generic.go:356] "Generic (PLEG): container finished" podID="012339dd-35cc-4f80-99d9-c9091d69f6df" containerID="c8fcf9f6053105c6a4256ee8d24eda2a475463b984899c59855b705cb6dc9127" exitCode=0 Oct 30 19:27:20.465651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.465338 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" event={"ID":"012339dd-35cc-4f80-99d9-c9091d69f6df","Type":"ContainerDied","Data":"c8fcf9f6053105c6a4256ee8d24eda2a475463b984899c59855b705cb6dc9127"} Oct 30 19:27:20.635027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.635003 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:20.697510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.697467 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\") pod \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " Oct 30 19:27:20.697689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.697550 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\" (UniqueName: \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\") pod \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " Oct 30 19:27:20.697689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.697666 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\") pod \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\" (UID: \"08b0e10d-41af-454f-8e7c-fcc729e9fda7\") " Oct 30 19:27:20.697920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.697888 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca" pod "08b0e10d-41af-454f-8e7c-fcc729e9fda7" (UID: "08b0e10d-41af-454f-8e7c-fcc729e9fda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:20.698370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.698350 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:20.699871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.699838 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm" pod "08b0e10d-41af-454f-8e7c-fcc729e9fda7" (UID: "08b0e10d-41af-454f-8e7c-fcc729e9fda7"). InnerVolumeSpecName "kube-api-access-lbkxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:20.700663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.700639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca" pod "08b0e10d-41af-454f-8e7c-fcc729e9fda7" (UID: "08b0e10d-41af-454f-8e7c-fcc729e9fda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:20.762348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.762244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:20.799996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.799954 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkxm\" (UniqueName: \"kubernetes.io/projected/08b0e10d-41af-454f-8e7c-fcc729e9fda7-kube-api-access-lbkxm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:20.799996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:20.799996 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08b0e10d-41af-454f-8e7c-fcc729e9fda7-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:21.475867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.475824 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" event={"ID":"08b0e10d-41af-454f-8e7c-fcc729e9fda7","Type":"ContainerDied","Data":"cae809724f5aa2792a645b30384fa261b9c0e7ab092691ca2b43c748253d38b9"} Oct 30 19:27:21.476406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.475893 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae809724f5aa2792a645b30384fa261b9c0e7ab092691ca2b43c748253d38b9" Oct 30 19:27:21.476406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.475847 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7" Oct 30 19:27:21.644332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.644307 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:21.712819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.712774 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\" (UniqueName: \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\") pod \"012339dd-35cc-4f80-99d9-c9091d69f6df\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " Oct 30 19:27:21.713010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.712892 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\") pod \"012339dd-35cc-4f80-99d9-c9091d69f6df\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " Oct 30 19:27:21.713122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.713065 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\") pod \"012339dd-35cc-4f80-99d9-c9091d69f6df\" (UID: \"012339dd-35cc-4f80-99d9-c9091d69f6df\") " Oct 30 19:27:21.714066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.714038 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca" pod "012339dd-35cc-4f80-99d9-c9091d69f6df" (UID: "012339dd-35cc-4f80-99d9-c9091d69f6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:21.716481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.716447 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca" pod "012339dd-35cc-4f80-99d9-c9091d69f6df" (UID: "012339dd-35cc-4f80-99d9-c9091d69f6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:21.716667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.716646 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd" pod "012339dd-35cc-4f80-99d9-c9091d69f6df" (UID: "012339dd-35cc-4f80-99d9-c9091d69f6df"). InnerVolumeSpecName "kube-api-access-6lwhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:21.815418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.815382 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwhd\" (UniqueName: \"kubernetes.io/projected/012339dd-35cc-4f80-99d9-c9091d69f6df-kube-api-access-6lwhd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:21.815418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.815412 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:21.815418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:21.815423 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012339dd-35cc-4f80-99d9-c9091d69f6df-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:22.408174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:22.408138 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:22.497132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:22.490321 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" Oct 30 19:27:22.497132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:22.490336 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p" event={"ID":"012339dd-35cc-4f80-99d9-c9091d69f6df","Type":"ContainerDied","Data":"e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e"} Oct 30 19:27:22.497132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:22.490418 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e80aa60e9c29dd398e1fa98b8e7ec49d56dcfeaca253ca8e09242dcee4765a6e" Oct 30 19:27:24.007755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:24.007699 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:25.880193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:25.880155 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:27.722249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:27.722191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.034379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.034172 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:27:28.072237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.072185 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.181226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.181180 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:27:28.218221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.218177 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.230894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.230853 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.344325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.344277 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.359189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.344534 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:28.572116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.571421 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.651310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.651204 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.747470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.733293 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:28.759139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:28.754345 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:27:29.044535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.044440 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.058882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.047534 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:27:29.116262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.086080 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.118165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.116559 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:29.183223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.183179 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.190300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.190269 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.212341 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.219423 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.247017 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.247821 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.273869 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:27:29.318176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.273921 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:27:29.359588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.359545 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.369749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.369719 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:29.586654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.586540 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.595366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.595330 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:27:29.655763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.655718 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.657252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.657228 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:29.735228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.735181 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:29.907752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:29.907220 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.061773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.061082 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.184395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.184296 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.190112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.186615 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:27:30.296116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.296055 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.302801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.301498 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.343244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.342397 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:27:30.362494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.361384 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.614117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.589194 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.659631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.659572 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.747109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:30.740208 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.069735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.069280 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:27:31.134819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.124368 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:31.134819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.124443 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.194311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.194265 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.195820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.195787 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.282421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.214864 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.355176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.339992 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.356878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.356225 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.381355 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.381318 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:27:31.389521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.389471 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.427111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.412266 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:27:31.581191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.581147 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:27:31.613283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.595645 4990 cpu_manager.go:414] "containerMap: removing container" podUID="012339dd-35cc-4f80-99d9-c9091d69f6df" containerName="revoke-keys" Oct 30 19:27:32.036284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.601320 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:32.036284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:31.675520 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:32.041109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.038227 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012339dd-35cc-4f80-99d9-c9091d69f6df" containerName="revoke-keys" Oct 30 19:27:32.041109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.038267 4990 cpu_manager.go:414] "containerMap: removing container" podUID="08b0e10d-41af-454f-8e7c-fcc729e9fda7" containerName="revoke-keys" Oct 30 19:27:32.041109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.038276 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b0e10d-41af-454f-8e7c-fcc729e9fda7" containerName="revoke-keys" Oct 30 19:27:32.048746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.048079 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="012339dd-35cc-4f80-99d9-c9091d69f6df" containerName="revoke-keys" Oct 30 19:27:32.053110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.050050 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b0e10d-41af-454f-8e7c-fcc729e9fda7" containerName="revoke-keys" Oct 30 19:27:32.095250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.094372 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:32.095250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.094525 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:32.118411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.118379 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:27:32.118670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.118651 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:32.118830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.118815 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:27:32.153111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.134459 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:27:32.153111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.129466 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.176113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.162874 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:27:32.176404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.176381 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:32.207213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.205927 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:27:32.229774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.229746 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:27:32.236737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.236707 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:27:32.279018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.278986 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:27:32.285713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.285002 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:32.285713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.279984 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286033 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286222 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286306 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286347 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286495 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286546 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286641 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.303323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.286717 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.387591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387546 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.387793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387716 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.387854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387826 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.387911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387876 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.387911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387906 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.388025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.387943 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.388123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.388072 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.388190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.388133 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.388245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.388191 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.388884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.388662 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.389004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.388935 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.389953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.389926 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.390348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.390327 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.391243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.391216 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.393009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.392970 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.393611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.393589 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.411990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.411939 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.414708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.414681 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.625942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.625858 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:32.968656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.968628 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:27:32.969516 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:32.969490 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c5e9ad_2f0a_42fe_8812_289bcec10cf7.slice/crio-6c5c8686a7c6d29958e11f3363fb4f2aa65c14932b87bfd5fe00c74886be7cf6 WatchSource:0}: Error finding container 6c5c8686a7c6d29958e11f3363fb4f2aa65c14932b87bfd5fe00c74886be7cf6: Status 404 returned error can't find the container with id 6c5c8686a7c6d29958e11f3363fb4f2aa65c14932b87bfd5fe00c74886be7cf6 Oct 30 19:27:32.972395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.972361 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:32.972475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.972428 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:32.972475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:32.972460 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:33.121977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:33.121943 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" event={"ID":"08c5e9ad-2f0a-42fe-8812-289bcec10cf7","Type":"ContainerStarted","Data":"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863"} Oct 30 19:27:33.122508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:33.122008 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" event={"ID":"08c5e9ad-2f0a-42fe-8812-289bcec10cf7","Type":"ContainerStarted","Data":"6c5c8686a7c6d29958e11f3363fb4f2aa65c14932b87bfd5fe00c74886be7cf6"} Oct 30 19:27:33.626282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:33.626252 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:34.033943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.033837 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podStartSLOduration=4.033821495 podStartE2EDuration="4.033821495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:33.166235421 +0000 UTC m=+1360.877760787" watchObservedRunningTime="2026-10-30 19:27:34.033821495 +0000 UTC m=+1361.745346864" Oct 30 19:27:34.035013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.034986 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7"] Oct 30 19:27:34.039441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.039422 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.058985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.058962 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7"] Oct 30 19:27:34.062326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.062265 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:34.062455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.062434 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:34.118181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.115554 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6r9q\" (UniqueName: \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.118181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.115764 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.118181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.115811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.216962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.216912 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.217539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.216969 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.217539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.217220 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6r9q\" (UniqueName: \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.218859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.218215 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.220869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.220843 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.235810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.235786 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6r9q\" (UniqueName: \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\") pod \"maas-api-revoke-sub-aace3d5b-zklr7\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.355699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.355663 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:34.628053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.627976 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:34.628053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.628027 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:34.708314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:34.708267 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7"] Oct 30 19:27:34.717193 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:34.717162 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c90e98_389f_4305_8b36_dc46102d7170.slice/crio-21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad WatchSource:0}: Error finding container 21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad: Status 404 returned error can't find the container with id 21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad Oct 30 19:27:35.055789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:35.055750 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:35.130764 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:27:35.130731 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:27:35.141295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:35.141270 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" event={"ID":"c6c90e98-389f-4305-8b36-dc46102d7170","Type":"ContainerStarted","Data":"21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad"} Oct 30 19:27:35.218856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:35.218830 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:35.627278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:35.627240 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:35.627482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:35.627299 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:36.151458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:36.151367 4990 generic.go:356] "Generic (PLEG): container finished" podID="c6c90e98-389f-4305-8b36-dc46102d7170" containerID="e6383711dfcaefdabd96cfa0a816486d84b066e4ef03ee124c2b6b5fa2a19d2e" exitCode=0 Oct 30 19:27:36.151612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:36.151455 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" event={"ID":"c6c90e98-389f-4305-8b36-dc46102d7170","Type":"ContainerDied","Data":"e6383711dfcaefdabd96cfa0a816486d84b066e4ef03ee124c2b6b5fa2a19d2e"} Oct 30 19:27:36.627101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:36.627053 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:36.627616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:36.627155 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:36.961703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:36.961623 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:37.405293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.405259 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:37.465020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.464989 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\") pod \"c6c90e98-389f-4305-8b36-dc46102d7170\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " Oct 30 19:27:37.465245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.465149 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\" (UniqueName: \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\") pod \"c6c90e98-389f-4305-8b36-dc46102d7170\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " Oct 30 19:27:37.465245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.465211 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\") pod \"c6c90e98-389f-4305-8b36-dc46102d7170\" (UID: \"c6c90e98-389f-4305-8b36-dc46102d7170\") " Oct 30 19:27:37.490896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.490853 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca" pod "c6c90e98-389f-4305-8b36-dc46102d7170" (UID: "c6c90e98-389f-4305-8b36-dc46102d7170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:37.493557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.493523 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q" pod "c6c90e98-389f-4305-8b36-dc46102d7170" (UID: "c6c90e98-389f-4305-8b36-dc46102d7170"). InnerVolumeSpecName "kube-api-access-n6r9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:37.494547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.494516 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca" pod "c6c90e98-389f-4305-8b36-dc46102d7170" (UID: "c6c90e98-389f-4305-8b36-dc46102d7170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:37.587115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.583232 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:37.587115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.583272 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c90e98-389f-4305-8b36-dc46102d7170-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:37.587115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.583289 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6r9q\" (UniqueName: \"kubernetes.io/projected/c6c90e98-389f-4305-8b36-dc46102d7170-kube-api-access-n6r9q\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:37.626522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.626485 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:37.626680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:37.626541 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:38.188115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.186989 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" event={"ID":"c6c90e98-389f-4305-8b36-dc46102d7170","Type":"ContainerDied","Data":"21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad"} Oct 30 19:27:38.188115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.187067 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b12c34ec163ca935d09c06cb72fd549342be71cd816b63e988666527955fad" Oct 30 19:27:38.188115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.187174 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7" Oct 30 19:27:38.626138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.626083 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:38.626324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.626143 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:38.960915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:38.960810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:39.626704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:39.626669 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:27:39.627177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:39.626734 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:27:40.630986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:40.630956 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:40.631514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:40.631494 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:40.632299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:40.632281 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:27:40.711789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:40.711725 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:42.311238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.311201 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:42.407542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.407509 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65"] Oct 30 19:27:42.408113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.408072 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c6c90e98-389f-4305-8b36-dc46102d7170" containerName="revoke-keys" Oct 30 19:27:42.408113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.408103 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c90e98-389f-4305-8b36-dc46102d7170" containerName="revoke-keys" Oct 30 19:27:42.408253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.408196 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c90e98-389f-4305-8b36-dc46102d7170" containerName="revoke-keys" Oct 30 19:27:42.411598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.411576 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.436025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.435979 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:42.436025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.436010 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:42.441991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.441967 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65"] Oct 30 19:27:42.565066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.564987 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mgh\" (UniqueName: \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.565254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.565130 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.565319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.565296 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.583693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.583659 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:42.666616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.666579 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.666826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.666706 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.666899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.666850 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mgh\" (UniqueName: \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.667530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.667499 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.669266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.669234 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.691362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.691333 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mgh\" (UniqueName: \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\") pod \"maas-api-revoke-sub-f07f9926-kzx65\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:42.725831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:42.725800 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:43.040389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:43.040362 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65"] Oct 30 19:27:43.041207 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:43.041183 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3401f092_7c60_4eaf_b44d_e84f803a41bf.slice/crio-63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904 WatchSource:0}: Error finding container 63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904: Status 404 returned error can't find the container with id 63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904 Oct 30 19:27:43.239156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:43.239121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" event={"ID":"3401f092-7c60-4eaf-b44d-e84f803a41bf","Type":"ContainerStarted","Data":"63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904"} Oct 30 19:27:44.216326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:44.216215 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:44.249925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:44.249893 4990 generic.go:356] "Generic (PLEG): container finished" podID="3401f092-7c60-4eaf-b44d-e84f803a41bf" containerID="47d0267d7e39981cfcf52f912684b6d9643a97250dcad00e7d05d87d1ccba4ff" exitCode=0 Oct 30 19:27:44.250127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:44.249936 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" event={"ID":"3401f092-7c60-4eaf-b44d-e84f803a41bf","Type":"ContainerDied","Data":"47d0267d7e39981cfcf52f912684b6d9643a97250dcad00e7d05d87d1ccba4ff"} Oct 30 19:27:45.415508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.415480 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:45.506772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.506740 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\") pod \"3401f092-7c60-4eaf-b44d-e84f803a41bf\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " Oct 30 19:27:45.506981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.506803 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\" (UniqueName: \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\") pod \"3401f092-7c60-4eaf-b44d-e84f803a41bf\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " Oct 30 19:27:45.506981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.506855 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\") pod \"3401f092-7c60-4eaf-b44d-e84f803a41bf\" (UID: \"3401f092-7c60-4eaf-b44d-e84f803a41bf\") " Oct 30 19:27:45.507878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.507350 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca" pod "3401f092-7c60-4eaf-b44d-e84f803a41bf" (UID: "3401f092-7c60-4eaf-b44d-e84f803a41bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:45.507878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.507706 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:45.509545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.509507 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca" pod "3401f092-7c60-4eaf-b44d-e84f803a41bf" (UID: "3401f092-7c60-4eaf-b44d-e84f803a41bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:45.510186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.510152 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh" pod "3401f092-7c60-4eaf-b44d-e84f803a41bf" (UID: "3401f092-7c60-4eaf-b44d-e84f803a41bf"). InnerVolumeSpecName "kube-api-access-s5mgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:45.609571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.609536 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mgh\" (UniqueName: \"kubernetes.io/projected/3401f092-7c60-4eaf-b44d-e84f803a41bf-kube-api-access-s5mgh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:45.609571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:45.609572 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3401f092-7c60-4eaf-b44d-e84f803a41bf-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:46.072889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.072829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:46.087173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.087138 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:27:46.088206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.088163 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3401f092-7c60-4eaf-b44d-e84f803a41bf" containerName="revoke-keys" Oct 30 19:27:46.088206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.088190 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3401f092-7c60-4eaf-b44d-e84f803a41bf" containerName="revoke-keys" Oct 30 19:27:46.088412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.088343 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3401f092-7c60-4eaf-b44d-e84f803a41bf" containerName="revoke-keys" Oct 30 19:27:46.094024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.094001 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.123086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.123052 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:27:46.218757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.218722 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.218912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.218860 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgh2\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.218976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.218914 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219009 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219039 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219146 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219173 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219189 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.219311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.219298 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.266105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.266072 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" Oct 30 19:27:46.266303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.266144 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65" event={"ID":"3401f092-7c60-4eaf-b44d-e84f803a41bf","Type":"ContainerDied","Data":"63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904"} Oct 30 19:27:46.266303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.266194 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63031f64e9695e06cb6fa7b023f62dd53c6bf9191411fe1a161018f4ad93b904" Oct 30 19:27:46.320391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320350 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgh2\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320412 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320461 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320479 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320558 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320595 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320621 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320730 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.320979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320903 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.321037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320980 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.321037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.320985 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.321165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.321127 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.321165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.321151 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.321511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.321489 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.323454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.323406 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.323537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.323456 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.337766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.337743 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.337941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.337922 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgh2\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.419681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.419653 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:46.746738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.746649 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:27:46.758182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.758144 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:46.758269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.758231 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:46.758322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:46.758273 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:27:47.298152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.289555 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" event={"ID":"9a274584-6f3e-49a8-a802-97179d42a05e","Type":"ContainerStarted","Data":"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31"} Oct 30 19:27:47.298152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.289624 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" event={"ID":"9a274584-6f3e-49a8-a802-97179d42a05e","Type":"ContainerStarted","Data":"b3902cea82ff68fcc6e625ed3bbd4258cad103fa27a501ed7e1f3489e0b57c4d"} Oct 30 19:27:47.348996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.348584 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podStartSLOduration=1.348567766 podStartE2EDuration="1.348567766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:47.337713713 +0000 UTC m=+1375.049239078" watchObservedRunningTime="2026-10-30 19:27:47.348567766 +0000 UTC m=+1375.060093133" Oct 30 19:27:47.420541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.420509 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:47.421754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.421724 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:47.421868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.421782 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:47.944079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:47.944038 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:48.465777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:48.465739 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:48.587114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:48.474100 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:48.907177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:48.907137 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65"] Oct 30 19:27:49.066294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.066241 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.094126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.093762 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.166233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.165770 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.215568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.203063 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.230123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.216105 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.230398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.230367 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffhd\" (UniqueName: \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.266130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.258055 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:27:49.285531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.265189 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 19:27:49.285829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.265273 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.285957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.283405 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65"] Oct 30 19:27:49.375115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.375030 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.375633 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.376867 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.376909 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffhd\" (UniqueName: \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.377062 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.378213 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.391365 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:49.401117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.391434 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:49.422247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.407572 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.453239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.453198 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:27:49.499602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.499562 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:49.500186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.500157 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:49.559753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.559462 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffhd\" (UniqueName: \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.598643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.598186 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.603115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.600063 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6622202d-wxs65\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.703165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.692286 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.720156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.719908 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:49.740344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.740244 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.934119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:49.933949 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 30 19:27:50.074115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.071182 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.074115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.072001 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:27:50.174261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.166015 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.174261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.166817 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:27:50.309119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.286940 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.371448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.325046 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:50.427279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.413730 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:50.439118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.431440 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:50.582152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.463195 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.582152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.464402 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.582152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.465581 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:50.582152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.465619 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:50.611131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.610621 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.634205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.633171 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:27:50.687116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.636206 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:50.687116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.636281 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:27:50.687116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.636324 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:27:50.687116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.636359 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:27:50.773133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.730725 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.777262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.744387 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.809163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.799122 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:50.809163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:50.799201 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:27:51.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.237471 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.239208 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.250461 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:27:51.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.299969 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.423293 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:51.422233 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361387d9_0cdf_43ac_880a_03a602c863df.slice/crio-d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f WatchSource:0}: Error finding container d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f: Status 404 returned error can't find the container with id d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f Oct 30 19:27:51.482116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.464215 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.491665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.491612 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:27:51.511383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.500236 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:51.586313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.586210 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:27:52.037593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.037484 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 19:27:52.041200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.936613 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 19:27:52.389218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.936659 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.418111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.971034 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:52.549076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.547951 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:52.549076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.024552 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:52.550340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.269206 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:52.550340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.549418 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.550340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.343721 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.550340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:51.675191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.611650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.306186 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.611650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.341283 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:27:52.669735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.669692 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:52.687390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.687338 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:52.863113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.688502 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:27:52.895357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:52.895309 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:27:53.041233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.041137 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:53.147070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.147024 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:53.668748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.668699 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:53.669430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.669396 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:53.669597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.669577 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:27:53.669755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.669737 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:27:53.669887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.669870 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:27:53.707503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.707462 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:53.781115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:53.777506 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:27:54.025103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.000166 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:54.025103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.000227 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:54.251145 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:27:54.207187 4990 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.764s" Oct 30 19:27:54.319115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.309611 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" event={"ID":"361387d9-0cdf-43ac-880a-03a602c863df","Type":"ContainerStarted","Data":"d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f"} Oct 30 19:27:54.424117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.423497 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:54.424117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.423554 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:54.517331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.517290 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65"] Oct 30 19:27:54.897126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.896630 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.936113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.925963 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.936113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.926345 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.936113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.926622 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.943068 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.943324 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.943412 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.943580 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.943610 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:27:54.946146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.944596 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:27:54.975116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.969736 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.986125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.985325 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:54.986125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:54.985392 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.340415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.340382 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:27:55.424113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.424042 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:55.424310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.424120 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:55.450062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.433327 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:27:55.456513 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.456480 4990 generic.go:356] "Generic (PLEG): container finished" podID="361387d9-0cdf-43ac-880a-03a602c863df" containerID="20d906e327d24a8edbae678d1cbe1f7a316ec90b1d29dca48fd4676273e82ed2" exitCode=0 Oct 30 19:27:55.456895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.456873 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" event={"ID":"361387d9-0cdf-43ac-880a-03a602c863df","Type":"ContainerDied","Data":"20d906e327d24a8edbae678d1cbe1f7a316ec90b1d29dca48fd4676273e82ed2"} Oct 30 19:27:55.471387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.468480 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:27:55.559388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.559357 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:27:55.624156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.566215 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:27:55.624156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.620404 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:27:55.642624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.642597 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:27:55.676059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.675216 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.727964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.727933 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:27:55.729110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.729034 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:27:55.748802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.748774 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:27:55.893166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:55.891990 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:56.421830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.421797 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:56.422314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.421859 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:56.758661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.758633 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:56.811087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.811051 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:56.842602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.842573 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\") pod \"361387d9-0cdf-43ac-880a-03a602c863df\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " Oct 30 19:27:56.842740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.842641 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\" (UniqueName: \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\") pod \"361387d9-0cdf-43ac-880a-03a602c863df\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " Oct 30 19:27:56.842803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.842779 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\") pod \"361387d9-0cdf-43ac-880a-03a602c863df\" (UID: \"361387d9-0cdf-43ac-880a-03a602c863df\") " Oct 30 19:27:56.843208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.843175 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca" pod "361387d9-0cdf-43ac-880a-03a602c863df" (UID: "361387d9-0cdf-43ac-880a-03a602c863df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:56.843474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.843451 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:56.844804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.844778 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd" pod "361387d9-0cdf-43ac-880a-03a602c863df" (UID: "361387d9-0cdf-43ac-880a-03a602c863df"). InnerVolumeSpecName "kube-api-access-2ffhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:56.844918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.844847 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca" pod "361387d9-0cdf-43ac-880a-03a602c863df" (UID: "361387d9-0cdf-43ac-880a-03a602c863df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:56.950204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.950170 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361387d9-0cdf-43ac-880a-03a602c863df-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:56.950450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:56.950434 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffhd\" (UniqueName: \"kubernetes.io/projected/361387d9-0cdf-43ac-880a-03a602c863df-kube-api-access-2ffhd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:27:57.420951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.420910 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:27:57.421126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.420971 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:27:57.497803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.497772 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" Oct 30 19:27:57.498233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.497769 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65" event={"ID":"361387d9-0cdf-43ac-880a-03a602c863df","Type":"ContainerDied","Data":"d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f"} Oct 30 19:27:57.498233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.497911 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23a6d5b2fcf78cf645b422e6feb08b5fd723bf9319b488ce190c4d76570a43f" Oct 30 19:27:57.722901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:57.722814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:58.523700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.523664 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj"] Oct 30 19:27:58.524645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.524623 4990 cpu_manager.go:414] "containerMap: removing container" podUID="361387d9-0cdf-43ac-880a-03a602c863df" containerName="revoke-keys" Oct 30 19:27:58.524696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.524649 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361387d9-0cdf-43ac-880a-03a602c863df" containerName="revoke-keys" Oct 30 19:27:58.524832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.524820 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="361387d9-0cdf-43ac-880a-03a602c863df" containerName="revoke-keys" Oct 30 19:27:58.529309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.529286 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.550395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.550365 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:58.553328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.553309 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj"] Oct 30 19:27:58.555760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.555736 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:27:58.555966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.555948 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:58.681372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.681332 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.681568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.681521 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.681692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.681674 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchk7\" (UniqueName: \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.782631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.782524 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.782819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.782688 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.782897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.782870 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchk7\" (UniqueName: \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.783356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.783332 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.785218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.785196 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.801290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.801257 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchk7\" (UniqueName: \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\") pod \"maas-api-revoke-sub-baa97429-c92xj\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:58.850008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:58.849981 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:27:59.161294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.161252 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj"] Oct 30 19:27:59.161756 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:27:59.161729 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813d44f2_c20c_42b4_8bf5_0c71f8eb94f4.slice/crio-b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a WatchSource:0}: Error finding container b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a: Status 404 returned error can't find the container with id b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a Oct 30 19:27:59.345111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.344160 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:59.464289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.464207 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:59.464289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.464264 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:59.519149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.519118 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" event={"ID":"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4","Type":"ContainerStarted","Data":"b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a"} Oct 30 19:27:59.532883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.532194 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:59.532883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.532699 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:27:59.534384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:27:59.534361 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:28:00.529860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:00.529777 4990 generic.go:356] "Generic (PLEG): container finished" podID="813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" containerID="8cc3c8554e41743925ca7b11016c449b8eaea3f259976f3f35c12db76ee55a3d" exitCode=0 Oct 30 19:28:00.530000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:00.529863 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" event={"ID":"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4","Type":"ContainerDied","Data":"8cc3c8554e41743925ca7b11016c449b8eaea3f259976f3f35c12db76ee55a3d"} Oct 30 19:28:01.142914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.142861 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:01.685057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.685023 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:28:01.831283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.831228 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\") pod \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " Oct 30 19:28:01.831529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.831312 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\" (UniqueName: \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\") pod \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " Oct 30 19:28:01.831572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.831550 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\") pod \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\" (UID: \"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4\") " Oct 30 19:28:01.832536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.832493 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca" pod "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" (UID: "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:01.834511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.834480 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca" pod "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" (UID: "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:01.834654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.834637 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7" pod "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" (UID: "813d44f2-c20c-42b4-8bf5-0c71f8eb94f4"). InnerVolumeSpecName "kube-api-access-mchk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:01.933885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.933847 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:01.933885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.933883 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:01.933885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:01.933894 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mchk7\" (UniqueName: \"kubernetes.io/projected/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4-kube-api-access-mchk7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:02.355413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.355378 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq"] Oct 30 19:28:02.356048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.356032 4990 cpu_manager.go:414] "containerMap: removing container" podUID="813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" containerName="revoke-keys" Oct 30 19:28:02.356120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.356050 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" containerName="revoke-keys" Oct 30 19:28:02.356170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.356160 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" containerName="revoke-keys" Oct 30 19:28:02.360180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.360162 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.383454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.383423 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq"] Oct 30 19:28:02.540848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.540795 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.541029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.541000 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.541113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.541064 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f524r\" (UniqueName: \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.548332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.548309 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" Oct 30 19:28:02.548468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.548343 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj" event={"ID":"813d44f2-c20c-42b4-8bf5-0c71f8eb94f4","Type":"ContainerDied","Data":"b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a"} Oct 30 19:28:02.548468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.548386 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1b3f72b27ee2716935d5058b18221e4b56436a03a0d4c19eacce8d9c16b132a" Oct 30 19:28:02.642154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.642068 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.642308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.642211 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.642308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.642249 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f524r\" (UniqueName: \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.642920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.642896 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.644697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.644672 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.660029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.660008 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f524r\" (UniqueName: \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\") pod \"maas-api-revoke-sub-e5b78259-p7lgq\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.678104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.678068 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:02.998824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:02.998396 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq"] Oct 30 19:28:03.008446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:03.007665 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:03.011608 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:03.011578 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0be76c_4f11_4c95_9393_a414f92f1ac5.slice/crio-7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422 WatchSource:0}: Error finding container 7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422: Status 404 returned error can't find the container with id 7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422 Oct 30 19:28:03.558974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:03.558935 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" event={"ID":"8c0be76c-4f11-4c95-9393-a414f92f1ac5","Type":"ContainerStarted","Data":"774133175bf819fd02025426b28a2cdccb9736cb4d2ab741b0e9c016960b9d59"} Oct 30 19:28:03.559510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:03.559034 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" event={"ID":"8c0be76c-4f11-4c95-9393-a414f92f1ac5","Type":"ContainerStarted","Data":"7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422"} Oct 30 19:28:03.594124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:03.594054 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" podStartSLOduration=1.594041497 podStartE2EDuration="1.594041497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:03.592988345 +0000 UTC m=+1391.304513712" watchObservedRunningTime="2026-10-30 19:28:03.594041497 +0000 UTC m=+1391.305566909" Oct 30 19:28:04.550048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:04.550011 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:04.575704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:04.575665 4990 generic.go:356] "Generic (PLEG): container finished" podID="8c0be76c-4f11-4c95-9393-a414f92f1ac5" containerID="774133175bf819fd02025426b28a2cdccb9736cb4d2ab741b0e9c016960b9d59" exitCode=0 Oct 30 19:28:04.576162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:04.575779 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" event={"ID":"8c0be76c-4f11-4c95-9393-a414f92f1ac5","Type":"ContainerDied","Data":"774133175bf819fd02025426b28a2cdccb9736cb4d2ab741b0e9c016960b9d59"} Oct 30 19:28:05.102191 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:05.102150 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:28:05.748548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.748522 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:05.792075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.792041 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\") pod \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " Oct 30 19:28:05.792249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.792225 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\" (UniqueName: \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\") pod \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " Oct 30 19:28:05.792376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.792358 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\") pod \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\" (UID: \"8c0be76c-4f11-4c95-9393-a414f92f1ac5\") " Oct 30 19:28:05.792486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.792459 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca" pod "8c0be76c-4f11-4c95-9393-a414f92f1ac5" (UID: "8c0be76c-4f11-4c95-9393-a414f92f1ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:05.793117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.793075 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:05.795253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.795225 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca" pod "8c0be76c-4f11-4c95-9393-a414f92f1ac5" (UID: "8c0be76c-4f11-4c95-9393-a414f92f1ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:05.796481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.796452 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r" pod "8c0be76c-4f11-4c95-9393-a414f92f1ac5" (UID: "8c0be76c-4f11-4c95-9393-a414f92f1ac5"). InnerVolumeSpecName "kube-api-access-f524r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:05.894564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.894482 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0be76c-4f11-4c95-9393-a414f92f1ac5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:05.894564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:05.894507 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f524r\" (UniqueName: \"kubernetes.io/projected/8c0be76c-4f11-4c95-9393-a414f92f1ac5-kube-api-access-f524r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:06.250525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:06.247300 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:06.596599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:06.596565 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" Oct 30 19:28:06.596599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:06.596575 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq" event={"ID":"8c0be76c-4f11-4c95-9393-a414f92f1ac5","Type":"ContainerDied","Data":"7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422"} Oct 30 19:28:06.596835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:06.596665 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d90fb7f72d8ccee2884dac75473d618433bf2dfacbd64982966548033f10422" Oct 30 19:28:08.018749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.018709 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:08.035412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.035382 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6"] Oct 30 19:28:08.044837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.044815 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b3f06d-d95c6"] Oct 30 19:28:08.405159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.404004 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19090908-21bc-4663-a2f0-7e236a6f6978" path="/var/lib/kubelet/pods/19090908-21bc-4663-a2f0-7e236a6f6978/volumes" Oct 30 19:28:08.847844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.847811 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw"] Oct 30 19:28:08.848492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.848475 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8c0be76c-4f11-4c95-9393-a414f92f1ac5" containerName="revoke-keys" Oct 30 19:28:08.848547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.848493 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0be76c-4f11-4c95-9393-a414f92f1ac5" containerName="revoke-keys" Oct 30 19:28:08.848587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.848578 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0be76c-4f11-4c95-9393-a414f92f1ac5" containerName="revoke-keys" Oct 30 19:28:08.853643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.853623 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:08.873555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.873527 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw"] Oct 30 19:28:08.876121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.876075 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:28:08.876258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.876075 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:28:08.933312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.933279 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:08.933517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.933503 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:08.933564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:08.933530 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j57n\" (UniqueName: \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.038757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.038716 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.039327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.039294 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j57n\" (UniqueName: \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.039477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.039462 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.040709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.040683 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.044040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.044013 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.058337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.058309 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j57n\" (UniqueName: \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\") pod \"maas-api-revoke-sub-1f149a23-ltrqw\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.175637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.175558 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:09.481267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.481234 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw"] Oct 30 19:28:09.483461 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:09.483421 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f663a90_512e_484b_a5cd_d4ec8626dfdf.slice/crio-7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a WatchSource:0}: Error finding container 7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a: Status 404 returned error can't find the container with id 7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a Oct 30 19:28:09.599721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.599680 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:09.635880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:09.635845 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" event={"ID":"2f663a90-512e-484b-a5cd-d4ec8626dfdf","Type":"ContainerStarted","Data":"7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a"} Oct 30 19:28:10.650595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:10.650558 4990 generic.go:356] "Generic (PLEG): container finished" podID="2f663a90-512e-484b-a5cd-d4ec8626dfdf" containerID="dfb3f3a3c8153c00a6e6b413923836a669facab627b666abb47249b8cdf4acb0" exitCode=0 Oct 30 19:28:10.650959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:10.650641 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" event={"ID":"2f663a90-512e-484b-a5cd-d4ec8626dfdf","Type":"ContainerDied","Data":"dfb3f3a3c8153c00a6e6b413923836a669facab627b666abb47249b8cdf4acb0"} Oct 30 19:28:11.796186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:11.795407 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:28:11.838459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:11.838408 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:11.896222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:11.896174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:11.896875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:11.896494 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.063433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.063386 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:28:12.337351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.337243 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.497235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.497187 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.746171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.590533 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.776116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.601671 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:12.860192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.698250 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:12.896116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.698285 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:28:12.922720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.861366 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:28:12.924714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.904400 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:12.928035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.924958 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:12.928035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:12.925028 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:28:13.133119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.133062 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.167533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.167503 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:28:13.370510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.370412 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.393430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.374784 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:28:13.847147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.847086 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.063119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.965153 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.063119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.972332 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.063119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.026185 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.063119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.030654 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.063119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.031481 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:28:14.190199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.187299 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.199728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.192324 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.309329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.215842 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.371112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:13.969229 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:28:14.396270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.396238 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:28:14.397843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.397818 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:28:14.424116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.418362 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:14.424116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.392320 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.786075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.723787 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\") pod \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " Oct 30 19:28:14.786075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.723945 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\") pod \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " Oct 30 19:28:14.786075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.724114 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\" (UniqueName: \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\") pod \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\" (UID: \"2f663a90-512e-484b-a5cd-d4ec8626dfdf\") " Oct 30 19:28:14.887979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.871182 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.913152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.911805 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n" pod "2f663a90-512e-484b-a5cd-d4ec8626dfdf" (UID: "2f663a90-512e-484b-a5cd-d4ec8626dfdf"). InnerVolumeSpecName "kube-api-access-6j57n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:14.922126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.916602 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca" pod "2f663a90-512e-484b-a5cd-d4ec8626dfdf" (UID: "2f663a90-512e-484b-a5cd-d4ec8626dfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:14.966695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.966636 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.971333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.971290 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.972843 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:14.974805 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.055746 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.055795 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.055797 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.055830 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j57n\" (UniqueName: \"kubernetes.io/projected/2f663a90-512e-484b-a5cd-d4ec8626dfdf-kube-api-access-6j57n\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.054826 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca" pod "2f663a90-512e-484b-a5cd-d4ec8626dfdf" (UID: "2f663a90-512e-484b-a5cd-d4ec8626dfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.031958 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.016215 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" podUID="7b26ecda-af1c-4159-a872-4bdec846ad44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.15:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.031204 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.065173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.031238 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:28:15.090128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.084818 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" event={"ID":"2f663a90-512e-484b-a5cd-d4ec8626dfdf","Type":"ContainerDied","Data":"7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a"} Oct 30 19:28:15.103301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.103265 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cda4dcb3dcf06dc598f73476c7a212e466c67ca040fefb5efe4907796c6779a" Oct 30 19:28:15.113158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.090429 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw" Oct 30 19:28:15.183271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.183235 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f663a90-512e-484b-a5cd-d4ec8626dfdf-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:15.218118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.204258 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.367334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.367224 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.428264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.428209 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:28:15.438555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.438499 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.542885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.542795 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:15.717202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.717055 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.809113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.805510 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6"] Oct 30 19:28:15.836116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.810310 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2f663a90-512e-484b-a5cd-d4ec8626dfdf" containerName="revoke-keys" Oct 30 19:28:15.852187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.852146 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f663a90-512e-484b-a5cd-d4ec8626dfdf" containerName="revoke-keys" Oct 30 19:28:15.852728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.852699 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f663a90-512e-484b-a5cd-d4ec8626dfdf" containerName="revoke-keys" Oct 30 19:28:15.898117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.887682 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:15.997565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:15.997479 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.021371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.020579 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 19:28:16.047196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.047166 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:28:16.073115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.048169 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:28:16.073694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.050468 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6"] Oct 30 19:28:16.073914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.058348 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:28:16.075201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.058381 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:16.110118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.062565 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.110118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.062601 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:28:16.110118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.062621 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:28:16.136116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.112396 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.176120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.168856 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.180115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.176459 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lk7\" (UniqueName: \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.204116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.125339 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:16.204399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.204362 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.232217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.232169 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.297507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.283976 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.297507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.284112 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lk7\" (UniqueName: \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.297507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.284246 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.297507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.285165 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.349604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.343846 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lk7\" (UniqueName: \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.373220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.373182 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:16.431262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.431221 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.482525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.482477 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.797111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.789425 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:28:16.798148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.790641 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62efc1c8-wgzv6\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.807154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.799351 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:16.842895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.842855 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.858564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.858532 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:16.872541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.870968 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:28:16.880133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.873481 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:28:16.880133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.875031 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:28:16.935696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.935670 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:28:16.953077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.953047 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:28:16.954015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.953982 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:28:16.989118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.988355 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:28:16.999134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:16.995505 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:28:17.310222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:17.310183 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6"] Oct 30 19:28:17.315076 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:17.315042 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7562cbd_7cf6_4bd5_8468_2bac1599726e.slice/crio-7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40 WatchSource:0}: Error finding container 7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40: Status 404 returned error can't find the container with id 7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40 Oct 30 19:28:17.736820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:17.736781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" event={"ID":"c7562cbd-7cf6-4bd5-8468-2bac1599726e","Type":"ContainerStarted","Data":"b017fc42dc56734776864bf00f84364a6401579814e8d401e978500daf9b1011"} Oct 30 19:28:17.737001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:17.736846 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" event={"ID":"c7562cbd-7cf6-4bd5-8468-2bac1599726e","Type":"ContainerStarted","Data":"7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40"} Oct 30 19:28:17.770926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:17.770867 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" podStartSLOduration=3.770848919 podStartE2EDuration="3.770848919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:17.769827012 +0000 UTC m=+1405.481352383" watchObservedRunningTime="2026-10-30 19:28:17.770848919 +0000 UTC m=+1405.482374287" Oct 30 19:28:18.198153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:18.198116 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:18.746927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:18.746888 4990 generic.go:356] "Generic (PLEG): container finished" podID="c7562cbd-7cf6-4bd5-8468-2bac1599726e" containerID="b017fc42dc56734776864bf00f84364a6401579814e8d401e978500daf9b1011" exitCode=0 Oct 30 19:28:18.747402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:18.746968 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" event={"ID":"c7562cbd-7cf6-4bd5-8468-2bac1599726e","Type":"ContainerDied","Data":"b017fc42dc56734776864bf00f84364a6401579814e8d401e978500daf9b1011"} Oct 30 19:28:19.925712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:19.925677 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:19.931979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:19.926665 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:20.048798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.048709 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\") pod \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " Oct 30 19:28:20.048955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.048817 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\") pod \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " Oct 30 19:28:20.049020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.048984 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\" (UniqueName: \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\") pod \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\" (UID: \"c7562cbd-7cf6-4bd5-8468-2bac1599726e\") " Oct 30 19:28:20.049335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.049284 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca" pod "c7562cbd-7cf6-4bd5-8468-2bac1599726e" (UID: "c7562cbd-7cf6-4bd5-8468-2bac1599726e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:20.049796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.049770 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:20.051287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.051250 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca" pod "c7562cbd-7cf6-4bd5-8468-2bac1599726e" (UID: "c7562cbd-7cf6-4bd5-8468-2bac1599726e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:20.051728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.051710 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7" pod "c7562cbd-7cf6-4bd5-8468-2bac1599726e" (UID: "c7562cbd-7cf6-4bd5-8468-2bac1599726e"). InnerVolumeSpecName "kube-api-access-p9lk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:20.150812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.150776 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lk7\" (UniqueName: \"kubernetes.io/projected/c7562cbd-7cf6-4bd5-8468-2bac1599726e-kube-api-access-p9lk7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:20.150812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.150808 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7562cbd-7cf6-4bd5-8468-2bac1599726e-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:20.765290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.765207 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" event={"ID":"c7562cbd-7cf6-4bd5-8468-2bac1599726e","Type":"ContainerDied","Data":"7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40"} Oct 30 19:28:20.765290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.765245 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6" Oct 30 19:28:20.765290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:20.765284 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fac2ec841475fd310c2fc93b8506480090de00e766848d54241608f214f5d40" Oct 30 19:28:21.245784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:21.245749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:21.879383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:21.879333 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:22.295320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.295244 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj"] Oct 30 19:28:22.295870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.295849 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c7562cbd-7cf6-4bd5-8468-2bac1599726e" containerName="revoke-keys" Oct 30 19:28:22.295870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.295865 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7562cbd-7cf6-4bd5-8468-2bac1599726e" containerName="revoke-keys" Oct 30 19:28:22.296056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.295973 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7562cbd-7cf6-4bd5-8468-2bac1599726e" containerName="revoke-keys" Oct 30 19:28:22.299704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.299686 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.319794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.319770 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj"] Oct 30 19:28:22.321757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.321734 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:28:22.321876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.321803 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:28:22.382433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.382398 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85d9\" (UniqueName: \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.382590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.382458 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.382632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.382598 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.483923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.483875 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85d9\" (UniqueName: \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.484160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.483960 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.484160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.483992 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.484816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.484788 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.486561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.486535 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.500755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.500725 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85d9\" (UniqueName: \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\") pod \"maas-api-revoke-sub-02f5a955-5s9gj\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.613989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.613950 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:22.930500 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:22.930471 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef04b2e_ff69_45d7_87f3_b5d637e4a45e.slice/crio-13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d WatchSource:0}: Error finding container 13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d: Status 404 returned error can't find the container with id 13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d Oct 30 19:28:22.930652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:22.930506 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj"] Oct 30 19:28:23.594117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:23.593317 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.793764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:23.793727 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" event={"ID":"fef04b2e-ff69-45d7-87f3-b5d637e4a45e","Type":"ContainerStarted","Data":"42d36cbf2c5bf7322ca09f1ac557f4a13580cc2611eb7706a0cc0a463aa2695a"} Oct 30 19:28:23.794021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:23.793825 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" event={"ID":"fef04b2e-ff69-45d7-87f3-b5d637e4a45e","Type":"ContainerStarted","Data":"13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d"} Oct 30 19:28:23.836699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:23.836646 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" podStartSLOduration=1.836630277 podStartE2EDuration="1.836630277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:23.831417081 +0000 UTC m=+1411.542942446" watchObservedRunningTime="2026-10-30 19:28:23.836630277 +0000 UTC m=+1411.548155644" Oct 30 19:28:24.803225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:24.803194 4990 generic.go:356] "Generic (PLEG): container finished" podID="fef04b2e-ff69-45d7-87f3-b5d637e4a45e" containerID="42d36cbf2c5bf7322ca09f1ac557f4a13580cc2611eb7706a0cc0a463aa2695a" exitCode=0 Oct 30 19:28:24.803618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:24.803287 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" event={"ID":"fef04b2e-ff69-45d7-87f3-b5d637e4a45e","Type":"ContainerDied","Data":"42d36cbf2c5bf7322ca09f1ac557f4a13580cc2611eb7706a0cc0a463aa2695a"} Oct 30 19:28:25.172701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:25.172656 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:25.979559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:25.979533 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:26.141678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.141601 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\" (UniqueName: \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\") pod \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " Oct 30 19:28:26.141820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.141704 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\") pod \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " Oct 30 19:28:26.141863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.141833 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\") pod \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\" (UID: \"fef04b2e-ff69-45d7-87f3-b5d637e4a45e\") " Oct 30 19:28:26.142201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.142175 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca" pod "fef04b2e-ff69-45d7-87f3-b5d637e4a45e" (UID: "fef04b2e-ff69-45d7-87f3-b5d637e4a45e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:26.142537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.142517 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:26.143606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.143584 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9" pod "fef04b2e-ff69-45d7-87f3-b5d637e4a45e" (UID: "fef04b2e-ff69-45d7-87f3-b5d637e4a45e"). InnerVolumeSpecName "kube-api-access-h85d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:26.143760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.143736 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca" pod "fef04b2e-ff69-45d7-87f3-b5d637e4a45e" (UID: "fef04b2e-ff69-45d7-87f3-b5d637e4a45e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:26.243567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.243533 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h85d9\" (UniqueName: \"kubernetes.io/projected/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-kube-api-access-h85d9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:26.243567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.243566 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef04b2e-ff69-45d7-87f3-b5d637e4a45e-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:26.828118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.827265 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" event={"ID":"fef04b2e-ff69-45d7-87f3-b5d637e4a45e","Type":"ContainerDied","Data":"13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d"} Oct 30 19:28:26.828118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.827339 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b27fe5428dd1491a2fd033f337e360d720c1f5c25c9db738300a817099a03d" Oct 30 19:28:26.828118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.827428 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj" Oct 30 19:28:26.933709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:26.933656 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:28.027465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:28.027427 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd"] Oct 30 19:28:28.036269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:28.036240 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e145d-kk2dd"] Oct 30 19:28:28.403006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:28.402972 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698836e9-acde-4b8f-b89e-e0a768a9df0d" path="/var/lib/kubelet/pods/698836e9-acde-4b8f-b89e-e0a768a9df0d/volumes" Oct 30 19:28:28.489351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:28.489318 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:29.061928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.061892 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:29.983353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.983319 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9241-b9dcc0"] Oct 30 19:28:29.983967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.983952 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fef04b2e-ff69-45d7-87f3-b5d637e4a45e" containerName="revoke-keys" Oct 30 19:28:29.984031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.983968 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef04b2e-ff69-45d7-87f3-b5d637e4a45e" containerName="revoke-keys" Oct 30 19:28:29.984064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.984056 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef04b2e-ff69-45d7-87f3-b5d637e4a45e" containerName="revoke-keys" Oct 30 19:28:29.988280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:29.988251 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:30.005479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.005447 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9241-b9dcc0"] Oct 30 19:28:30.007806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.007781 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:28:30.092827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.092787 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfkd\" (UniqueName: \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\") pod \"test-curl-9241-b9dcc0\" (UID: \"0000c554-207e-4658-9f4b-2965d3d1194a\") " pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:30.196167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.195982 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfkd\" (UniqueName: \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\") pod \"test-curl-9241-b9dcc0\" (UID: \"0000c554-207e-4658-9f4b-2965d3d1194a\") " pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:30.223647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.223620 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfkd\" (UniqueName: \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\") pod \"test-curl-9241-b9dcc0\" (UID: \"0000c554-207e-4658-9f4b-2965d3d1194a\") " pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:30.239625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.239537 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:30.303130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.303084 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:30.611888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.611846 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9241-b9dcc0"] Oct 30 19:28:30.612539 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:30.612506 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0000c554_207e_4658_9f4b_2965d3d1194a.slice/crio-7576e28a96521ee48ac1fb3e0c5edcb55c648f8d74598640eea125bd88baa26e WatchSource:0}: Error finding container 7576e28a96521ee48ac1fb3e0c5edcb55c648f8d74598640eea125bd88baa26e: Status 404 returned error can't find the container with id 7576e28a96521ee48ac1fb3e0c5edcb55c648f8d74598640eea125bd88baa26e Oct 30 19:28:30.614550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.614532 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:28:30.863188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:30.863072 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9241-b9dcc0" event={"ID":"0000c554-207e-4658-9f4b-2965d3d1194a","Type":"ContainerStarted","Data":"7576e28a96521ee48ac1fb3e0c5edcb55c648f8d74598640eea125bd88baa26e"} Oct 30 19:28:31.253623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.253548 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:28:31.259471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.259444 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.294227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.293659 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:28:31.313651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313613 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.313849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313682 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.313849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313788 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpb8\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.313849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313827 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.314065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313852 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.314065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313886 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.314065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.313976 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.314065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.314037 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.314301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.314114 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.415723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415688 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.415915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415730 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.415915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415777 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.415915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415869 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415928 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.415986 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416163 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416217 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416329 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpb8\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416431 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416568 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416711 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416769 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.416871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.416855 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.418713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.418690 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.418828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.418724 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.434255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.434227 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.434467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.434436 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpb8\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.579232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.579196 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:31.928988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.927743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:28:31.937331 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:31.937305 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2995e2be_452e_40a3_9c38_164e786c2302.slice/crio-3f9546fa13709fd73b2d38112cd5f6de17f13db313db20d64d19a72f8b3f6b95 WatchSource:0}: Error finding container 3f9546fa13709fd73b2d38112cd5f6de17f13db313db20d64d19a72f8b3f6b95: Status 404 returned error can't find the container with id 3f9546fa13709fd73b2d38112cd5f6de17f13db313db20d64d19a72f8b3f6b95 Oct 30 19:28:31.940558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.940522 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:31.940646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.940601 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:31.940719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:31.940642 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:32.120115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.119344 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:32.709155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.707812 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9"] Oct 30 19:28:32.724183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.724142 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9"] Oct 30 19:28:32.724361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.724274 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.739545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.737556 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:28:32.748298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.746468 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:28:32.844027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.843995 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.844493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.844269 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.844493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.844336 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn94l\" (UniqueName: \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.927351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.927313 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" event={"ID":"2995e2be-452e-40a3-9c38-164e786c2302","Type":"ContainerStarted","Data":"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73"} Oct 30 19:28:32.927539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.927382 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" event={"ID":"2995e2be-452e-40a3-9c38-164e786c2302","Type":"ContainerStarted","Data":"3f9546fa13709fd73b2d38112cd5f6de17f13db313db20d64d19a72f8b3f6b95"} Oct 30 19:28:32.945676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.945645 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.946074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.946054 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.946594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.946573 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.946714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.946700 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn94l\" (UniqueName: \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:32.948669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:32.948648 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:33.013596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.013521 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn94l\" (UniqueName: \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\") pod \"maas-api-revoke-sub-9d2db0d4-gn2l9\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:33.014703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.014641 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podStartSLOduration=2.014618528 podStartE2EDuration="2.014618528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:32.964979117 +0000 UTC m=+1420.676504485" watchObservedRunningTime="2026-10-30 19:28:33.014618528 +0000 UTC m=+1420.726143903" Oct 30 19:28:33.046983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.046952 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:33.404728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.404694 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9"] Oct 30 19:28:33.409875 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:33.409845 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66436e1a_eca6_42a9_bfd3_4d1a570b212b.slice/crio-bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a WatchSource:0}: Error finding container bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a: Status 404 returned error can't find the container with id bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a Oct 30 19:28:33.586121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.580438 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:33.586121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.584192 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:33.586121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.584244 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:33.807565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.806019 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:33.937017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.936952 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9241-b9dcc0" event={"ID":"0000c554-207e-4658-9f4b-2965d3d1194a","Type":"ContainerStarted","Data":"7ed387f1f07f910909ea3c7601abd303bd5b898240e7b1b0beb2700098c793a4"} Oct 30 19:28:33.940702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.939981 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" event={"ID":"66436e1a-eca6-42a9-bfd3-4d1a570b212b","Type":"ContainerStarted","Data":"df623b9b373f015bb0376f7c3b3b27ea0a8e47445037cd83d4c93a15bf34a2dd"} Oct 30 19:28:33.940702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.940027 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" event={"ID":"66436e1a-eca6-42a9-bfd3-4d1a570b212b","Type":"ContainerStarted","Data":"bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a"} Oct 30 19:28:33.993492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:33.993424 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9241-b9dcc0" podStartSLOduration=1.979409289 podStartE2EDuration="4.99340397s" totalImagesPullingTime="3.013994681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:33.971243514 +0000 UTC m=+1421.682768882" watchObservedRunningTime="2026-10-30 19:28:33.99340397 +0000 UTC m=+1421.704929338" Oct 30 19:28:34.019123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.019038 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" podStartSLOduration=2.0190257 podStartE2EDuration="2.0190257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:34.014038539 +0000 UTC m=+1421.725563912" watchObservedRunningTime="2026-10-30 19:28:34.0190257 +0000 UTC m=+1421.730551110" Oct 30 19:28:34.582115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.581590 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:34.582115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.581667 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:34.910194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.905348 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9241-b9dcc0"] Oct 30 19:28:34.910678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.910457 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9241-b9dcc0"] Oct 30 19:28:34.951849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.951815 4990 generic.go:356] "Generic (PLEG): container finished" podID="66436e1a-eca6-42a9-bfd3-4d1a570b212b" containerID="df623b9b373f015bb0376f7c3b3b27ea0a8e47445037cd83d4c93a15bf34a2dd" exitCode=0 Oct 30 19:28:34.952050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:34.951915 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" event={"ID":"66436e1a-eca6-42a9-bfd3-4d1a570b212b","Type":"ContainerDied","Data":"df623b9b373f015bb0376f7c3b3b27ea0a8e47445037cd83d4c93a15bf34a2dd"} Oct 30 19:28:35.055504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.054963 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9241-7445af"] Oct 30 19:28:35.055916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.055887 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" containerName="test-curl-9241-b9dcc0" Oct 30 19:28:35.055916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.055917 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" containerName="test-curl-9241-b9dcc0" Oct 30 19:28:35.056221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.056200 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" containerName="test-curl-9241-b9dcc0" Oct 30 19:28:35.062372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.062236 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:35.150235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.150145 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9241-7445af"] Oct 30 19:28:35.208889 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:35.208799 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:28:35.220290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.220258 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\") pod \"test-curl-9241-7445af\" (UID: \"577b02f4-03da-4d49-99fb-6465ffd60f8c\") " pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:35.323116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.322082 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\") pod \"test-curl-9241-7445af\" (UID: \"577b02f4-03da-4d49-99fb-6465ffd60f8c\") " pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:35.343113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.341781 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\") pod \"test-curl-9241-7445af\" (UID: \"577b02f4-03da-4d49-99fb-6465ffd60f8c\") " pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:35.447141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.446323 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:35.590114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.586498 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:35.590114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.586576 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:35.750553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.750464 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:35.821046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.821011 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w"] Oct 30 19:28:35.831662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.831632 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:35.841105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.841055 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w"] Oct 30 19:28:35.860321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.858730 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9241-7445af"] Oct 30 19:28:35.951610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.951331 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qlt\" (UniqueName: \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:35.952342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.951649 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:35.952342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:35.951698 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.004133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.004031 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9241-7445af" event={"ID":"577b02f4-03da-4d49-99fb-6465ffd60f8c","Type":"ContainerStarted","Data":"e39b149c304e93674da5650413952823c7cedb24c453ccd9751a220d51a06c05"} Oct 30 19:28:36.004680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.004179 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9241-b9dcc0" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" containerName="test-curl-9241-b9dcc0" containerID="cri-o://7ed387f1f07f910909ea3c7601abd303bd5b898240e7b1b0beb2700098c793a4" gracePeriod=2 Oct 30 19:28:36.015351 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:36.015321 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-b9dcc0\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:36.053198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.053167 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.053319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.053218 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.053378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.053341 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qlt\" (UniqueName: \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.054298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.054272 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.060748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.060702 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.104868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.103011 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qlt\" (UniqueName: \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\") pod \"maas-api-revoke-sub-b0636b68-wd52w\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.351115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.342617 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:36.441162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.440295 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:36.453007 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:36.452484 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-b9dcc0\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:36.485129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.480610 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\") pod \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " Oct 30 19:28:36.485129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.480715 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\") pod \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " Oct 30 19:28:36.485129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.480870 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\" (UniqueName: \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\") pod \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\" (UID: \"66436e1a-eca6-42a9-bfd3-4d1a570b212b\") " Oct 30 19:28:36.519122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.515199 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l" pod "66436e1a-eca6-42a9-bfd3-4d1a570b212b" (UID: "66436e1a-eca6-42a9-bfd3-4d1a570b212b"). InnerVolumeSpecName "kube-api-access-bn94l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:36.519122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.515564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca" pod "66436e1a-eca6-42a9-bfd3-4d1a570b212b" (UID: "66436e1a-eca6-42a9-bfd3-4d1a570b212b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:36.531665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.531609 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca" pod "66436e1a-eca6-42a9-bfd3-4d1a570b212b" (UID: "66436e1a-eca6-42a9-bfd3-4d1a570b212b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:36.616156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.584372 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:36.616156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.584410 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66436e1a-eca6-42a9-bfd3-4d1a570b212b-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:36.616156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.584430 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn94l\" (UniqueName: \"kubernetes.io/projected/66436e1a-eca6-42a9-bfd3-4d1a570b212b-kube-api-access-bn94l\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:36.616156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.615224 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:36.616156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:36.615299 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:37.719603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:37.717568 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:37.719603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:37.718281 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:37.946268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:37.946212 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:37.979705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:37.979590 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.064840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.062518 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.125506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.125451 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.147570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.147529 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:28:38.205116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.194497 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w"] Oct 30 19:28:38.238167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.236238 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9241-7445af" event={"ID":"577b02f4-03da-4d49-99fb-6465ffd60f8c","Type":"ContainerStarted","Data":"e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1"} Oct 30 19:28:38.274910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.274867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" event={"ID":"66436e1a-eca6-42a9-bfd3-4d1a570b212b","Type":"ContainerDied","Data":"bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a"} Oct 30 19:28:38.275234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.275207 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc9e83f7eea88b9866507df4a1b50d33c1c2ffabbe16a68bb993b914614ca61a" Oct 30 19:28:38.275471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.275453 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9" Oct 30 19:28:38.413409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.374512 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:28:38.413409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.386212 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:38.551193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.506846 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:38.551193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.512966 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:28:38.551193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.544174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:28:38.604106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.600170 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9241-7445af" podStartSLOduration=3.6001475 podStartE2EDuration="3.6001475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:38.374447748 +0000 UTC m=+1426.085973115" watchObservedRunningTime="2026-10-30 19:28:38.6001475 +0000 UTC m=+1426.311672868" Oct 30 19:28:38.631214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.631168 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:38.631388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.631235 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:38.930192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.913646 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:39.052116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.947599 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:39.052116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:38.961243 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:28:39.052116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.002411 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:39.052116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.003066 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:28:39.160040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.068193 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.160040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.068648 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:28:39.160040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.127271 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.160040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.127678 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:39.162207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.162166 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.162768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.162746 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:28:39.233248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.215156 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.265870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.265825 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.454466 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.464607 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.470466 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.470533 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.472555 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:39.655180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.472601 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.726737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.726693 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:28:39.731212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.731174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" podUID="7b26ecda-af1c-4159-a872-4bdec846ad44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.15:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.841149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.749772 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:28:39.841149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.779208 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:39.841149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.784161 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.858173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.841841 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:39.858173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.841906 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.858173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.843385 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx"] Oct 30 19:28:39.944488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.944392 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b4937e-5lzqx"] Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.971351 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:39.971412 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.003124 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.003429 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.004286 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:28:40.083161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.004343 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:28:40.146643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.109425 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:40.198220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.173542 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.226117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.222084 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.226450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.226391 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.250273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.250224 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.281117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.276339 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.281117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.276402 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.281593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.281564 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.281744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.281722 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.342686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.342643 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:28:40.369116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.353505 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.491308 4990 generic.go:356] "Generic (PLEG): container finished" podID="0000c554-207e-4658-9f4b-2965d3d1194a" containerID="7ed387f1f07f910909ea3c7601abd303bd5b898240e7b1b0beb2700098c793a4" exitCode=137 Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.516268 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.516620 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.516884 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.517019 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.517210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.517052 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.602592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.602544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" event={"ID":"e4bce047-2cb6-479a-91e6-c0fde409e640","Type":"ContainerStarted","Data":"5a2863122f13ecd78de4b5d2814bf725e3747a018e613d4c066b1299a0a129d6"} Oct 30 19:28:40.628290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.628243 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:40.628585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.628541 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:40.771168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.727351 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.926887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:40.926850 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f524915f-1d34-4521-a6be-b488178c1b78" path="/var/lib/kubelet/pods/f524915f-1d34-4521-a6be-b488178c1b78/volumes" Oct 30 19:28:41.053476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.009319 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:41.053476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.016814 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.153152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.150084 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:41.195470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.184132 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.257663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.252705 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.451200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.385646 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.451200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.385953 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.451200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.417398 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:41.527118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.508571 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\" (UniqueName: \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\") pod \"0000c554-207e-4658-9f4b-2965d3d1194a\" (UID: \"0000c554-207e-4658-9f4b-2965d3d1194a\") " Oct 30 19:28:41.527118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.525773 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.528945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.528068 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:41.568713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.568671 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 19:28:41.668115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.598864 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:41.705122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.598910 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:41.705122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.680525 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:41.850181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.849849 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd" pod "0000c554-207e-4658-9f4b-2965d3d1194a" (UID: "0000c554-207e-4658-9f4b-2965d3d1194a"). InnerVolumeSpecName "kube-api-access-gnfkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:41.882918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.866531 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:28:41.928460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:41.914261 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 19:28:42.418121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.051199 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.768050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.051465 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.770163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.051503 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:28:42.784756 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.213377 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.784929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.213561 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:28:42.784929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.367532 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.785059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.508008 4990 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:42.785059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.785024 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.785196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.638751 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.785196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.541896 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:28:42.785342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.662964 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:28:42.785342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.785274 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:28:42.785342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.784244 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.811995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.811951 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfkd\" (UniqueName: \"kubernetes.io/projected/0000c554-207e-4658-9f4b-2965d3d1194a-kube-api-access-gnfkd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:42.875016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.874969 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.975753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.975717 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0000c554-207e-4658-9f4b-2965d3d1194a" path="/var/lib/kubelet/pods/0000c554-207e-4658-9f4b-2965d3d1194a/volumes" Oct 30 19:28:42.976882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.976788 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:28:42.977105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.977076 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:42.977279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.977245 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:28:42.984803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.984779 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:28:42.986499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.986475 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:28:42.986635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.986529 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:28:42.990398 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:42.990371 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0000c554_207e_4658_9f4b_2965d3d1194a.slice/crio-conmon-7ed387f1f07f910909ea3c7601abd303bd5b898240e7b1b0beb2700098c793a4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:28:42.996746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:42.996728 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:28:43.000535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.000425 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:28:43.005822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.005805 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-b9dcc0" Oct 30 19:28:43.015136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.007469 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:28:43.015243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.015168 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:28:43.015689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.007483 4990 scope.go:122] "RemoveContainer" containerID="7ed387f1f07f910909ea3c7601abd303bd5b898240e7b1b0beb2700098c793a4" Oct 30 19:28:43.020608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.020381 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:28:43.423478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.423441 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9241-7445af"] Oct 30 19:28:43.423987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.423637 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9241-7445af" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" containerName="test-curl-9241-7445af" containerID="cri-o://e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1" gracePeriod=2 Oct 30 19:28:43.431563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.427752 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9241-7445af"] Oct 30 19:28:43.450119 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:43.434945 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:43.772930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.772854 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:43.773442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.773418 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:43.774488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:43.774461 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:28:43.784279 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:43.784251 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:43.872728 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:43.869441 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:44.015861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.015827 4990 generic.go:356] "Generic (PLEG): container finished" podID="e4bce047-2cb6-479a-91e6-c0fde409e640" containerID="e20827e44232284ed19496acaf7b144efd8dfe5c25ce4801c630e18c6361822c" exitCode=0 Oct 30 19:28:44.016057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.015921 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" event={"ID":"e4bce047-2cb6-479a-91e6-c0fde409e640","Type":"ContainerDied","Data":"e20827e44232284ed19496acaf7b144efd8dfe5c25ce4801c630e18c6361822c"} Oct 30 19:28:44.186409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.186379 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:44.197575 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:44.197522 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:44.265123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.265068 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\") pod \"e4bce047-2cb6-479a-91e6-c0fde409e640\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " Oct 30 19:28:44.265268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.265204 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\") pod \"e4bce047-2cb6-479a-91e6-c0fde409e640\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " Oct 30 19:28:44.265333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.265303 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\" (UniqueName: \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\") pod \"e4bce047-2cb6-479a-91e6-c0fde409e640\" (UID: \"e4bce047-2cb6-479a-91e6-c0fde409e640\") " Oct 30 19:28:44.265847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.265814 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca" pod "e4bce047-2cb6-479a-91e6-c0fde409e640" (UID: "e4bce047-2cb6-479a-91e6-c0fde409e640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:44.267520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.267487 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt" pod "e4bce047-2cb6-479a-91e6-c0fde409e640" (UID: "e4bce047-2cb6-479a-91e6-c0fde409e640"). InnerVolumeSpecName "kube-api-access-k9qlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:44.267655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.267633 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca" pod "e4bce047-2cb6-479a-91e6-c0fde409e640" (UID: "e4bce047-2cb6-479a-91e6-c0fde409e640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:44.367838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.367761 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:44.367838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.367799 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qlt\" (UniqueName: \"kubernetes.io/projected/e4bce047-2cb6-479a-91e6-c0fde409e640-kube-api-access-k9qlt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:44.367838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:44.367810 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4bce047-2cb6-479a-91e6-c0fde409e640-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:45.029493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.029463 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" Oct 30 19:28:45.029987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.029505 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w" event={"ID":"e4bce047-2cb6-479a-91e6-c0fde409e640","Type":"ContainerDied","Data":"5a2863122f13ecd78de4b5d2814bf725e3747a018e613d4c066b1299a0a129d6"} Oct 30 19:28:45.029987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.029559 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2863122f13ecd78de4b5d2814bf725e3747a018e613d4c066b1299a0a129d6" Oct 30 19:28:45.253437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.253399 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:45.664259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.664232 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:45.688979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.688948 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\") pod \"577b02f4-03da-4d49-99fb-6465ffd60f8c\" (UID: \"577b02f4-03da-4d49-99fb-6465ffd60f8c\") " Oct 30 19:28:45.691442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.691410 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb" pod "577b02f4-03da-4d49-99fb-6465ffd60f8c" (UID: "577b02f4-03da-4d49-99fb-6465ffd60f8c"). InnerVolumeSpecName "kube-api-access-czfnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:45.803996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:45.803962 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfnb\" (UniqueName: \"kubernetes.io/projected/577b02f4-03da-4d49-99fb-6465ffd60f8c-kube-api-access-czfnb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:46.041011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.040977 4990 generic.go:356] "Generic (PLEG): container finished" podID="577b02f4-03da-4d49-99fb-6465ffd60f8c" containerID="e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1" exitCode=137 Oct 30 19:28:46.041512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.041027 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:46.041512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.041037 4990 scope.go:122] "RemoveContainer" containerID="e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1" Oct 30 19:28:46.053210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.053183 4990 scope.go:122] "RemoveContainer" containerID="e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1" Oct 30 19:28:46.053568 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:46.053531 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1\": container with ID starting with e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1 not found: ID does not exist" containerID="e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1" Oct 30 19:28:46.053659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.053572 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1"} err="failed to get container status \"e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1\": rpc error: code = NotFound desc = could not find container \"e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1\": container with ID starting with e9b26b081ac5582cf2eec163d0bceb12c7785fe07e3f1b18b7934bbdd4a752f1 not found: ID does not exist" Oct 30 19:28:46.062306 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:46.062253 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:46.121216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.121187 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:46.123330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.123306 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s"] Oct 30 19:28:46.124006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.123987 4990 cpu_manager.go:414] "containerMap: removing container" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" containerName="test-curl-9241-7445af" Oct 30 19:28:46.124050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124008 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" containerName="test-curl-9241-7445af" Oct 30 19:28:46.124050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124020 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e4bce047-2cb6-479a-91e6-c0fde409e640" containerName="revoke-keys" Oct 30 19:28:46.124050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124025 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bce047-2cb6-479a-91e6-c0fde409e640" containerName="revoke-keys" Oct 30 19:28:46.124050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124044 4990 cpu_manager.go:414] "containerMap: removing container" podUID="66436e1a-eca6-42a9-bfd3-4d1a570b212b" containerName="revoke-keys" Oct 30 19:28:46.124050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124049 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66436e1a-eca6-42a9-bfd3-4d1a570b212b" containerName="revoke-keys" Oct 30 19:28:46.124219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124169 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" containerName="test-curl-9241-7445af" Oct 30 19:28:46.124219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124178 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bce047-2cb6-479a-91e6-c0fde409e640" containerName="revoke-keys" Oct 30 19:28:46.124219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.124185 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="66436e1a-eca6-42a9-bfd3-4d1a570b212b" containerName="revoke-keys" Oct 30 19:28:46.128946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.128928 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.140293 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:28:46.140256 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9241-7445af\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" pod="opendatahub/test-curl-9241-7445af" Oct 30 19:28:46.144300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.144280 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s"] Oct 30 19:28:46.144812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.144793 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:28:46.145711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.145692 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:28:46.212755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.212712 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwpl\" (UniqueName: \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.221153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.213032 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.221153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.213172 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.301610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.301568 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:46.315317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.315246 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.315454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.315359 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwpl\" (UniqueName: \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.315664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.315646 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.316687 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.316488 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.318878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.318850 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.337767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.337738 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwpl\" (UniqueName: \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\") pod \"maas-api-revoke-sub-a09429ac-8j67s\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.401616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.401582 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577b02f4-03da-4d49-99fb-6465ffd60f8c" path="/var/lib/kubelet/pods/577b02f4-03da-4d49-99fb-6465ffd60f8c/volumes" Oct 30 19:28:46.447432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.447405 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:46.760956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.760919 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s"] Oct 30 19:28:46.762070 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:46.762044 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf939a409_1671_4646_9219_63d9d9b6f337.slice/crio-f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d WatchSource:0}: Error finding container f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d: Status 404 returned error can't find the container with id f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d Oct 30 19:28:46.789729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.789700 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:46.861914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.861881 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:28:46.869130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.869071 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.908663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.908629 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d4245aa3-openshift-default-dockercfg-4n9kd\"" Oct 30 19:28:46.919694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.919659 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:28:46.925295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.925265 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv92x\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.931442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.931411 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.931715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.931695 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.931851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.931836 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.932004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.931989 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.932166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.932148 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.932317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.932302 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.932429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.932415 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:46.932665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:46.932645 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.032789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.032685 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:47.033811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033784 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.033931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033829 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.033931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033859 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.033931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033895 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033941 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.033979 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034014 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034173 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034252 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv92x\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034295 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034312 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.034773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.034753 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.035123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.035066 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.035123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.035102 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.036878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.036859 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.036973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.036954 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.061063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.058282 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.061063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.058435 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" event={"ID":"f939a409-1671-4646-9219-63d9d9b6f337","Type":"ContainerStarted","Data":"f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d"} Oct 30 19:28:47.061063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.060378 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv92x\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\") pod \"e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.213962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.213923 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:47.284745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.284625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:47.430227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.430186 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:47.553516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.553488 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:28:47.554723 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:47.554681 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b41e2fb_05b2_4784_bd95_02b4094975ce.slice/crio-8d6d9dc9dfe3117115dbcabddb121ffbede19c85db654283ef5d2294714d8269 WatchSource:0}: Error finding container 8d6d9dc9dfe3117115dbcabddb121ffbede19c85db654283ef5d2294714d8269: Status 404 returned error can't find the container with id 8d6d9dc9dfe3117115dbcabddb121ffbede19c85db654283ef5d2294714d8269 Oct 30 19:28:47.557382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.557344 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:47.557472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.557418 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:47.557472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:47.557446 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:48.073230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.073189 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerStarted","Data":"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e"} Oct 30 19:28:48.073705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.073247 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerStarted","Data":"8d6d9dc9dfe3117115dbcabddb121ffbede19c85db654283ef5d2294714d8269"} Oct 30 19:28:48.074850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.074821 4990 generic.go:356] "Generic (PLEG): container finished" podID="f939a409-1671-4646-9219-63d9d9b6f337" containerID="f132208c8b949ead689f8c0a21df3cae145d74b537abb3d84484d9dcfe8fd30c" exitCode=0 Oct 30 19:28:48.074954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.074898 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" event={"ID":"f939a409-1671-4646-9219-63d9d9b6f337","Type":"ContainerDied","Data":"f132208c8b949ead689f8c0a21df3cae145d74b537abb3d84484d9dcfe8fd30c"} Oct 30 19:28:48.114493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.114442 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podStartSLOduration=2.114429108 podStartE2EDuration="2.114429108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:48.109983617 +0000 UTC m=+1435.821508982" watchObservedRunningTime="2026-10-30 19:28:48.114429108 +0000 UTC m=+1435.825954477" Oct 30 19:28:48.214905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.214820 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:48.220538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:48.220518 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:49.033049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.033007 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6"] Oct 30 19:28:49.043246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.043211 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c219201c-8cqs6"] Oct 30 19:28:49.084210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.083648 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:49.084684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.084662 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:28:49.213008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.212970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:49.334564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.334530 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:49.354148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.354117 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:28:49.354793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.354773 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f939a409-1671-4646-9219-63d9d9b6f337" containerName="revoke-keys" Oct 30 19:28:49.354893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.354795 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f939a409-1671-4646-9219-63d9d9b6f337" containerName="revoke-keys" Oct 30 19:28:49.354988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.354975 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f939a409-1671-4646-9219-63d9d9b6f337" containerName="revoke-keys" Oct 30 19:28:49.358608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.358587 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.372961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.372944 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:28:49.373044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.372951 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:28:49.373044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.373009 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4lz7b\"" Oct 30 19:28:49.375479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.375463 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:28:49.386346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.386320 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:28:49.480026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.479991 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\" (UniqueName: \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\") pod \"f939a409-1671-4646-9219-63d9d9b6f337\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " Oct 30 19:28:49.480256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\") pod \"f939a409-1671-4646-9219-63d9d9b6f337\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " Oct 30 19:28:49.480350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480329 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\") pod \"f939a409-1671-4646-9219-63d9d9b6f337\" (UID: \"f939a409-1671-4646-9219-63d9d9b6f337\") " Oct 30 19:28:49.480641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480618 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.480767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480651 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca" pod "f939a409-1671-4646-9219-63d9d9b6f337" (UID: "f939a409-1671-4646-9219-63d9d9b6f337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:49.480767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480757 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.480946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480932 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.481004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480964 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.481004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.480993 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.481165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.481143 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49kr\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.481238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.481176 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.481294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.481265 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:49.482708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.482685 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca" pod "f939a409-1671-4646-9219-63d9d9b6f337" (UID: "f939a409-1671-4646-9219-63d9d9b6f337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:49.482798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.482740 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl" pod "f939a409-1671-4646-9219-63d9d9b6f337" (UID: "f939a409-1671-4646-9219-63d9d9b6f337"). InnerVolumeSpecName "kube-api-access-ktwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:49.581972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.581944 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582005 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49kr\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582046 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582174 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582273 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582476 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582521 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.582631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582602 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktwpl\" (UniqueName: \"kubernetes.io/projected/f939a409-1671-4646-9219-63d9d9b6f337-kube-api-access-ktwpl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:49.582631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.582619 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f939a409-1671-4646-9219-63d9d9b6f337-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:28:49.583169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.583141 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.584473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.584445 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.584554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.584494 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.584598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.584584 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.584732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.584710 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.600028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.600003 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.600195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.600178 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49kr\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nrwm\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.672763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.672731 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:49.788917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.788876 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:49.963902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:49.963810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:50.021253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.021221 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:28:50.021392 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:28:50.021231 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e757b0d_22db_4c33_ad55_c3a6f9711735.slice/crio-5fae26694fd735b82622cf1ccdcc23ad5fac855d95cb6192a7c68f3ab636a92a WatchSource:0}: Error finding container 5fae26694fd735b82622cf1ccdcc23ad5fac855d95cb6192a7c68f3ab636a92a: Status 404 returned error can't find the container with id 5fae26694fd735b82622cf1ccdcc23ad5fac855d95cb6192a7c68f3ab636a92a Oct 30 19:28:50.093129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.093075 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" Oct 30 19:28:50.093129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.093120 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s" event={"ID":"f939a409-1671-4646-9219-63d9d9b6f337","Type":"ContainerDied","Data":"f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d"} Oct 30 19:28:50.093634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.093179 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24a32a36d32592b52c5a5c6d863e45635802fb97fa9145e62a26b862651a83d" Oct 30 19:28:50.094726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.094702 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" event={"ID":"7e757b0d-22db-4c33-ad55-c3a6f9711735","Type":"ContainerStarted","Data":"5fae26694fd735b82622cf1ccdcc23ad5fac855d95cb6192a7c68f3ab636a92a"} Oct 30 19:28:50.403135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:50.403080 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d06973b-436d-4b9e-a980-af02ca2f1204" path="/var/lib/kubelet/pods/9d06973b-436d-4b9e-a980-af02ca2f1204/volumes" Oct 30 19:28:51.216474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:51.216435 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:51.521688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:51.521605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:51.712116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:51.711562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:52.552297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:52.552260 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:52.788753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:52.788703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:52.917518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:52.917479 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:52.917639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:52.917551 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:28:52.992827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:52.992757 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.030768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.030734 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq"] Oct 30 19:28:53.040223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.040196 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75479acf-dlndq"] Oct 30 19:28:53.147173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.147134 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" event={"ID":"7e757b0d-22db-4c33-ad55-c3a6f9711735","Type":"ContainerStarted","Data":"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254"} Oct 30 19:28:53.147680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.147506 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.147957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.147938 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:28:53.150167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.150136 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:53.150340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.150315 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:53.193286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.193169 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podStartSLOduration=1.300586573 podStartE2EDuration="4.193150615s" totalImagesPullingTime="2.892564042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:53.186252325 +0000 UTC m=+1440.897777693" watchObservedRunningTime="2026-10-30 19:28:53.193150615 +0000 UTC m=+1440.904675984" Oct 30 19:28:53.467242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.467152 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.653112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:53.648814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:54.156270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:54.156238 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:54.156440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:54.156299 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:54.402441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:54.402409 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8af8c92-6761-409c-85fd-1f6f4fd151f2" path="/var/lib/kubelet/pods/a8af8c92-6761-409c-85fd-1f6f4fd151f2/volumes" Oct 30 19:28:54.929528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:54.929484 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:56.675466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:56.673672 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:57.156395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:57.156355 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:57.156569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:57.156435 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:58.314957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:58.314912 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:58.522672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:58.522633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:58.693602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:58.693507 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:59.115038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:59.114999 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:59.287833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:28:59.287796 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:00.156317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:00.156279 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:00.156704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:00.156352 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:00.640102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:00.639793 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:01.763121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:01.761229 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.910126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:01.908435 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.919257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:01.919200 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.924240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:01.924181 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 19:29:01.934244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:01.934193 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:02.002404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.002355 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:29:02.039117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.038996 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.062137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.056023 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.084164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.084134 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.164391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.164352 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:29:02.185681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.182713 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250498 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250558 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxks\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250618 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250647 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250670 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250740 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250779 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250890 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.250962 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.281116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.261235 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355402 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355468 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355573 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355639 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355767 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxks\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355832 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355870 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.355894 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.357105 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.357385 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.363114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.358022 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.366241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.366200 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.383767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.376460 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.384521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.384489 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.390431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.390387 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.392688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.392659 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.520899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.520851 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.521958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.521935 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxks\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.522576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.522549 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.535020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.534990 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:02.766789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.765230 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.768781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.767530 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:02.962153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.947973 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.962153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.949462 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:29:02.987141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.976163 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.987141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:02.976651 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:03.057415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.048692 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.057415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.049232 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:03.101237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.066173 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.101237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.066767 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:03.209278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.184787 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.210076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.210049 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:03.287124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.256950 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:03.342264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.342174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:03.346115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.266323 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.350125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.346654 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:03.372379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.371162 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.373258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.373234 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:03.445474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.438240 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm"] Oct 30 19:29:03.536301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.536247 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.884176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.831447 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.884176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:03.844389 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5bb6ff7-z74wm"] Oct 30 19:29:04.004137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.001068 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.030200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.030155 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:04.059783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.051783 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.068348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.068306 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.212010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.210890 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:29:04.236346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.216676 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:04.236346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.216768 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:04.236346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.216813 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:04.362997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.362881 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.390225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.390180 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.558579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.558533 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:04.560566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.560533 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:29:04.883117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:04.880247 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:29:05.012425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.005436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:05.012425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.006825 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:29:05.027926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.015760 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Oct 30 19:29:05.033874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.033124 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.078119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.073359 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.084263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.084174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.430170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.426379 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.439812 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.440036 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.617126 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.517971 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380c5dcf-74cd-426e-a640-a3252bc7b694" path="/var/lib/kubelet/pods/380c5dcf-74cd-426e-a640-a3252bc7b694/volumes" Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:05.617901 4990 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.225s" Oct 30 19:29:05.635454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.618171 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:05.662721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.475082 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.693071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.591536 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.709259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.591385 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.727193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.727161 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:05.949329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.949228 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.953424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:05.953325 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.009396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.009342 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.086850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.086666 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:06.143540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.136426 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.149126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.148031 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.165638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.164822 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:06.165638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.164873 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:06.165638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.165262 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.168627 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:06.167614 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:29:06.178415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.172212 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:06.226171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.206045 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:06.350940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.350914 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:06.355194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.355157 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:06.538165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.537728 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" event={"ID":"7c27d140-38d9-44f1-8537-fb9b30b4aa54","Type":"ContainerStarted","Data":"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340"} Oct 30 19:29:06.538165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.537788 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" event={"ID":"7c27d140-38d9-44f1-8537-fb9b30b4aa54","Type":"ContainerStarted","Data":"5c9dd2f84f86dced6258386575e37ea02f2a2173b67f533da3215fd583a660da"} Oct 30 19:29:06.538165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.537843 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:06.538165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.537865 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:29:06.538165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.537910 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:06.541980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.541954 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:06.542468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.542441 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:06.542737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.542714 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:06.576397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.575709 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:06.578632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.578577 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:29:06.582827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.582790 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:06.677117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.676453 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podStartSLOduration=5.676436608 podStartE2EDuration="5.676436608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:06.674726175 +0000 UTC m=+1454.386251541" watchObservedRunningTime="2026-10-30 19:29:06.676436608 +0000 UTC m=+1454.387961976" Oct 30 19:29:06.837463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:06.837427 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:07.105300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:07.103777 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:07.535901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:07.535809 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:07.536048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:07.535892 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:07.838669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:07.838627 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:08.012057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:08.012007 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:08.535480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:08.535448 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:08.535911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:08.535504 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:09.029710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.029671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:09.158064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.158030 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:09.158257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.158080 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:09.200112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.199238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:09.538274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.538230 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:09.539159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:09.539120 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:10.149136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:10.149071 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:10.536010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:10.535841 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:10.536010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:10.535928 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:10.928957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:10.928919 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:11.197750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:11.197648 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:11.536290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:11.536185 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:11.536290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:11.536240 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:12.156369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:12.156338 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:12.156857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:12.156437 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:12.333111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:12.324631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:12.535877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:12.535778 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:29:12.535877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:12.535839 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:29:13.540471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:13.540442 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:13.540935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:13.540912 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:13.541720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:13.541702 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:29:14.030178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:14.030147 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj"] Oct 30 19:29:14.040355 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:14.040320 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:14.054962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:14.054927 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57eba0bf-fnwmj"] Oct 30 19:29:14.402545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:14.402506 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d648c22d-6c5b-47f4-b7b1-43d52052af2d" path="/var/lib/kubelet/pods/d648c22d-6c5b-47f4-b7b1-43d52052af2d/volumes" Oct 30 19:29:15.156049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:15.156016 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:15.156464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:15.156078 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:15.539388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:15.539306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:15.707641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:15.707594 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:17.497583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:17.497501 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:17.820712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:17.820608 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:18.158014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:18.157446 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:29:18.158014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:18.157549 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:29:19.176854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.176811 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:29:19.211272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.211182 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 19:29:19.394220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.394171 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:19.584221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.584174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:29:19.651288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.651242 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cc594" podUID="7b26ecda-af1c-4159-a872-4bdec846ad44" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.15:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:19.929166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:19.928161 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.034202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.034157 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:20.091214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.091170 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.106208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.106166 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:29:20.179949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.179167 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:29:20.181988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.180701 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:20.355798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.355730 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.403277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.403188 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.537006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.536920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:20.579983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.579947 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:20.580187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.580030 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:20.588376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.588341 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.597241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:20.597214 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:21.189676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.189630 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:21.289631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.289602 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:21.321527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.321500 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:29:21.400107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.399052 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:21.400107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.399207 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:21.400107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.399250 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:21.698189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:21.698145 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:23.477536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:23.477499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:25.030302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:25.030218 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn"] Oct 30 19:29:25.039994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:25.039959 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e8643-x94rn"] Oct 30 19:29:25.090342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:25.090305 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:26.402101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:26.402068 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f12094-d639-46a9-a531-6c31d5e9b830" path="/var/lib/kubelet/pods/c5f12094-d639-46a9-a531-6c31d5e9b830/volumes" Oct 30 19:29:26.778525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:26.778434 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:26.933854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:26.933812 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:27.080058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:27.080025 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:28.639727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:28.639682 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:29.034972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:29.034885 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk"] Oct 30 19:29:29.044701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:29.044674 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb1bfe6-cqdxk"] Oct 30 19:29:30.405072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:30.405037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:30.410370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:30.410339 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a5706b-b468-428b-bdfd-92ee5faf2055" path="/var/lib/kubelet/pods/39a5706b-b468-428b-bdfd-92ee5faf2055/volumes" Oct 30 19:29:30.584302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:30.584264 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:30.742153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:30.742053 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:31.294615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:31.294561 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:29:31.294924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:31.294892 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" containerID="cri-o://d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79" gracePeriod=30 Oct 30 19:29:33.326239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.326187 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:33.348228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.348174 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:33.353230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.353194 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:33.366353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.366309 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:33.391257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.391208 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:33.455244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.455191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:33.771691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.771601 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:33.840540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:33.840494 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.009244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.009194 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.043292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.043187 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.091714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.091671 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.337225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.336253 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.337225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.336959 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:34.359123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.357340 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:34.359123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.357907 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:34.375116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.360565 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.375116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.360788 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:34.375116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.374227 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.375116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.374664 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:34.414132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.413172 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:29:34.489116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.485238 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:34.516227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.516184 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.525116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.516717 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:35.014151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.984971 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:29:35.014151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:34.988389 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:35.017482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.015676 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.017482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.016813 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Oct 30 19:29:35.035561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.018027 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:35.087324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.087277 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.126060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.117769 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.142507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.142466 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:29:35.142817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.142801 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:35.340116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.336833 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.420253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.420211 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:35.429112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.420822 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.430124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.430074 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:35.510080 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.485038 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.542476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.542431 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.544736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.544708 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:35.544951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.544926 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.666182 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:35.665868 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:29:35.752321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.751551 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.999256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:35.988169 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.030842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.029178 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:29:36.030842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.029914 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.191157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.139675 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.191157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.140119 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:36.451442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.449595 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:36.451442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.449924 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:36.535151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.522222 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:29:36.535151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.522631 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:36.545629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.545582 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.546223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.546198 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.573186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.570548 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:36.803540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.758717 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:36.824150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.820352 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:36.845143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.832474 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:36.979162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.966430 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.982873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:36.982847 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:37.021144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.017782 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:37.067609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.067521 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:29:37.117812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.117776 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:29:37.119619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.119597 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:37.120585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.120565 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 19:29:37.120837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.120815 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:37.126447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.125155 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:37.430709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.430682 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:29:37.599637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599588 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599651 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599714 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599760 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599836 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599893 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.599930 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.600077 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.600482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.600212 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\") pod \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\" (UID: \"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3\") " Oct 30 19:29:37.601896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.601606 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:37.601896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.601639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:37.601896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.601859 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:37.602358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.602310 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:37.602682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.602645 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:37.605066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.604562 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:37.605066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.604897 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "kube-api-access-bbjc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:37.605663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.605633 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:29:37.606072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.606047 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token" pod "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" (UID: "1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:37.626421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.626390 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerID="d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79" exitCode=0 Oct 30 19:29:37.626698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.626682 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" event={"ID":"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3","Type":"ContainerDied","Data":"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79"} Oct 30 19:29:37.626746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.626735 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" event={"ID":"1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3","Type":"ContainerDied","Data":"c082df2530a005c8b47baf360111c03b84129387a5c211cc3b59cfe947ed110f"} Oct 30 19:29:37.626778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.626767 4990 scope.go:122] "RemoveContainer" containerID="d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79" Oct 30 19:29:37.626951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.626937 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np" Oct 30 19:29:37.641270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.641250 4990 scope.go:122] "RemoveContainer" containerID="d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79" Oct 30 19:29:37.641560 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:37.641541 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79\": container with ID starting with d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79 not found: ID does not exist" containerID="d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79" Oct 30 19:29:37.641617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.641568 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79"} err="failed to get container status \"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79\": rpc error: code = NotFound desc = could not find container \"d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79\": container with ID starting with d410bfc08a0db509e3b7bf30b2ec9ceddf95c6319d81feba852e431ec6ed7b79 not found: ID does not exist" Oct 30 19:29:37.712341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.712245 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715536 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715564 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715573 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715583 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715591 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715599 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715607 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715617 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.715996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.715626 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbjc7\" (UniqueName: \"kubernetes.io/projected/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3-kube-api-access-bbjc7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:37.718716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:37.718684 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-vq9np"] Oct 30 19:29:38.406152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:38.406111 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" path="/var/lib/kubelet/pods/1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3/volumes" Oct 30 19:29:38.612005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:38.611969 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:38.751837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:38.751742 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:40.450142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:40.450085 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:42.274776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:42.274737 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:43.687751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.687711 4990 generic.go:356] "Generic (PLEG): container finished" podID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerID="23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e" exitCode=0 Oct 30 19:29:43.688315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.687782 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerDied","Data":"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e"} Oct 30 19:29:43.688388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.688370 4990 scope.go:122] "RemoveContainer" containerID="23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e" Oct 30 19:29:43.690043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.690016 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:43.690141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.690083 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:43.690141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:43.690129 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:44.098113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:44.097510 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:44.716946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:44.716906 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerStarted","Data":"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f"} Oct 30 19:29:45.214523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:45.214483 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:29:45.215668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:45.215639 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.215814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:45.215707 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:45.738079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:45.738029 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:46.215582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.215547 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.215903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.215855 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:46.708598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.708563 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:29:46.709240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.709225 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" Oct 30 19:29:46.709294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.709241 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" Oct 30 19:29:46.709358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.709348 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f764c77-0f46-4a4d-ba19-8f7f3cbbc7d3" containerName="istio-proxy" Oct 30 19:29:46.712223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.712206 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763159 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763199 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763225 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdlvt\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763257 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763484 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763529 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.763694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763638 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.764029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763778 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.764029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.763883 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.812492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.812446 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:29:46.903446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.902782 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.903812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.903781 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.903984 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904020 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdlvt\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904080 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904137 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904172 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904223 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904299 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904716 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.904747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.904735 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.905462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.905432 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.905554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.905498 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.907234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.907210 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.909381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.909359 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.925599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.925571 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:46.925873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:46.925855 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdlvt\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:47.026012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.025924 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:47.134461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.134423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:47.241191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.220026 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.241191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.220118 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:47.293276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.292159 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:47.432220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.432192 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:29:47.435006 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:29:47.434978 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff60e5c7_be31_4317_bff1_44e17babd616.slice/crio-e7910bcc90c77455c3b8881f4613137e9d04805cabb7065e0c94dd8500a91cec WatchSource:0}: Error finding container e7910bcc90c77455c3b8881f4613137e9d04805cabb7065e0c94dd8500a91cec: Status 404 returned error can't find the container with id e7910bcc90c77455c3b8881f4613137e9d04805cabb7065e0c94dd8500a91cec Oct 30 19:29:47.437690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.437658 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:47.437758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.437724 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:47.437795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.437755 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:29:47.725047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.724997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:47.795116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.794148 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" event={"ID":"ff60e5c7-be31-4317-bff1-44e17babd616","Type":"ContainerStarted","Data":"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab"} Oct 30 19:29:47.795116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.794215 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" event={"ID":"ff60e5c7-be31-4317-bff1-44e17babd616","Type":"ContainerStarted","Data":"e7910bcc90c77455c3b8881f4613137e9d04805cabb7065e0c94dd8500a91cec"} Oct 30 19:29:47.860120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:47.856248 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podStartSLOduration=1.856227759 podStartE2EDuration="1.856227759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:47.85384526 +0000 UTC m=+1495.565370627" watchObservedRunningTime="2026-10-30 19:29:47.856227759 +0000 UTC m=+1495.567753125" Oct 30 19:29:48.028399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:48.028251 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:48.031625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:48.031583 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.031799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:48.031650 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:48.215531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:48.215486 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.215731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:48.215549 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:49.041116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.032545 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:49.041116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.032622 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:49.087112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.085905 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:29:49.214113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.213653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:49.219361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.218459 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:49.219361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.218514 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:49.535956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:49.535910 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:50.042115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.029717 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:50.045911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.042720 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:50.232115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.231521 4990 patch_prober.go:29] interesting pod/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:29:50.232115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.231585 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 19:29:50.400116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.399512 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:50.596421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:50.596385 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.037115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.027904 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:51.037115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.027977 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:51.058138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.047554 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.100126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.096636 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:29:51.100126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.096895 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" containerID="cri-o://7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f" gracePeriod=30 Oct 30 19:29:51.243822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.242917 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.552121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.551161 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 19:29:51.598953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.598269 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:51.758966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.758933 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.763733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.763694 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:51.777216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.777176 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:51.893296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.893188 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:51.968116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.943382 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.990235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:51.990191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:52.024231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.024180 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:29:52.027427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.027396 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:52.048219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.048172 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:52.091596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.091550 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.147279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.147179 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:52.458167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.427577 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:52.592115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.591734 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:52.622443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.620697 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.622443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.620961 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:52.622443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.621402 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:52.871163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.871114 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:29:52.871975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:52.871953 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:53.018653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.018605 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.027412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.027378 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:53.167411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.159338 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.167411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.161629 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:53.167411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.161946 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:29:53.203696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.203648 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 19:29:53.238117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.205050 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:53.238420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.238381 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:53.320024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.319707 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:53.320542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.320517 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:29:53.468848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.467280 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:29:53.479888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.475283 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:53.503144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.495173 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.503144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.495626 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:53.645737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.645689 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.647450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.647414 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 19:29:53.649676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.649654 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:53.652434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.652408 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:29:53.668258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.656200 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:29:53.718104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.703787 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:29:53.831194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.831156 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:29:53.888471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.888438 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 19:29:53.920110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.915060 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 19:29:53.957115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:53.953285 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:29:54.002153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.000678 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:29:54.012460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.011173 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:29:54.013928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.013902 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:29:54.027918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.027883 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:54.028071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.027942 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:54.326652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.326622 4990 scope.go:122] "RemoveContainer" containerID="49d66f0ca5e110ca8ae6f2d407183cda053f6701aca5dcda891e573abcde7028" Oct 30 19:29:54.339180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.339151 4990 scope.go:122] "RemoveContainer" containerID="f4c0d5e01b69eaa8748358f51b9a5b755aa874eb192ca98254f3c9a17c084773" Oct 30 19:29:54.357669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.357643 4990 scope.go:122] "RemoveContainer" containerID="eec48e5767547db0bbdc9bd66fed6549a75061da84cb0bcfddf6cd2942ed36f3" Oct 30 19:29:54.405818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.405783 4990 scope.go:122] "RemoveContainer" containerID="9016240cb06f219852e83e17215a33b265357d92eb241aeae5063d25a4169dc7" Oct 30 19:29:54.419998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.419954 4990 scope.go:122] "RemoveContainer" containerID="2e00b3638f403453f3bb12ab710b7ffb39a40179e19912c3c21055a27fe0a4da" Oct 30 19:29:54.425450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.425416 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.815059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.815024 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks"] Oct 30 19:29:54.819377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.819358 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:54.833527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.833498 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:54.843424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.843399 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:29:54.866122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.858770 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks"] Oct 30 19:29:54.917159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.915593 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4rp\" (UniqueName: \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:54.917159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.915816 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:54.917159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.915971 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:54.934510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:54.934340 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.017859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.017824 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.018049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.018009 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.018149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.018137 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4rp\" (UniqueName: \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.019138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.019112 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.020758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.020732 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.026866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.026836 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:55.026988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.026904 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:55.036023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.035984 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4rp\" (UniqueName: \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\") pod \"maas-api-revoke-sub-ba6f29aa-mxzks\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.147501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.147426 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:55.487636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.487606 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks"] Oct 30 19:29:55.488199 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:29:55.488175 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf37ff8a_875d_4405_9fe6_b2bfb757d6cb.slice/crio-63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b WatchSource:0}: Error finding container 63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b: Status 404 returned error can't find the container with id 63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b Oct 30 19:29:55.526156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:55.526121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" event={"ID":"df37ff8a-875d-4405-9fe6-b2bfb757d6cb","Type":"ContainerStarted","Data":"63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b"} Oct 30 19:29:56.026991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.026952 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:29:56.027193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.027018 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 19:29:56.540738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.540699 4990 generic.go:356] "Generic (PLEG): container finished" podID="df37ff8a-875d-4405-9fe6-b2bfb757d6cb" containerID="117e6ecd120422c7ece88583f676582335be7bf0c9685480b19bdb1f05c1f65c" exitCode=0 Oct 30 19:29:56.541264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.540781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" event={"ID":"df37ff8a-875d-4405-9fe6-b2bfb757d6cb","Type":"ContainerDied","Data":"117e6ecd120422c7ece88583f676582335be7bf0c9685480b19bdb1f05c1f65c"} Oct 30 19:29:56.708470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.708150 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:56.905320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:56.905064 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.003629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.003601 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:29:57.054261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054206 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054306 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054368 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054413 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054612 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:57.054734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054673 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054753 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054804 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.054832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054821 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:57.054983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054844 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:57.054983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.054899 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\") pod \"2b41e2fb-05b2-4784-bd95-02b4094975ce\" (UID: \"2b41e2fb-05b2-4784-bd95-02b4094975ce\") " Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055393 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055486 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055785 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b41e2fb-05b2-4784-bd95-02b4094975ce-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055806 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055821 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055834 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.056264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.055847 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.057210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.057086 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:57.057618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.057594 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:57.057708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.057616 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "kube-api-access-qv92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:57.057944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.057917 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo" pod "2b41e2fb-05b2-4784-bd95-02b4094975ce" (UID: "2b41e2fb-05b2-4784-bd95-02b4094975ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:29:57.157251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.157213 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.157251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.157252 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.157485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.157266 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b41e2fb-05b2-4784-bd95-02b4094975ce-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.157485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.157275 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv92x\" (UniqueName: \"kubernetes.io/projected/2b41e2fb-05b2-4784-bd95-02b4094975ce-kube-api-access-qv92x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.554519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.554482 4990 generic.go:356] "Generic (PLEG): container finished" podID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerID="7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f" exitCode=0 Oct 30 19:29:57.555028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.554850 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" Oct 30 19:29:57.555125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.555017 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerDied","Data":"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f"} Oct 30 19:29:57.555125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.555081 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j" event={"ID":"2b41e2fb-05b2-4784-bd95-02b4094975ce","Type":"ContainerDied","Data":"8d6d9dc9dfe3117115dbcabddb121ffbede19c85db654283ef5d2294714d8269"} Oct 30 19:29:57.555230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.555137 4990 scope.go:122] "RemoveContainer" containerID="7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f" Oct 30 19:29:57.569947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.569922 4990 scope.go:122] "RemoveContainer" containerID="23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e" Oct 30 19:29:57.579615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.579591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.584380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.584354 4990 scope.go:122] "RemoveContainer" containerID="7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f" Oct 30 19:29:57.584712 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:57.584681 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f\": container with ID starting with 7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f not found: ID does not exist" containerID="7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f" Oct 30 19:29:57.584799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.584725 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f"} err="failed to get container status \"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f\": rpc error: code = NotFound desc = could not find container \"7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f\": container with ID starting with 7923d3f6b72270792311c272fc00fdf95df338790b76cc739a2dca3e4ad8ee3f not found: ID does not exist" Oct 30 19:29:57.584799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.584751 4990 scope.go:122] "RemoveContainer" containerID="23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e" Oct 30 19:29:57.585069 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:29:57.585031 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e\": container with ID starting with 23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e not found: ID does not exist" containerID="23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e" Oct 30 19:29:57.585248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.585078 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e"} err="failed to get container status \"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e\": rpc error: code = NotFound desc = could not find container \"23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e\": container with ID starting with 23c3a6df6d87b7ac82324f1e39b0155070c1febbf6a581c2959b177e28dc2c6e not found: ID does not exist" Oct 30 19:29:57.616227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.616194 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:29:57.623322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.623294 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d4245aa3-openshift-default-5c5f469fb5-sg22j"] Oct 30 19:29:57.726543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.726493 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.747739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.747714 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:57.767038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.767001 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\" (UniqueName: \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\") pod \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " Oct 30 19:29:57.767201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.767041 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\") pod \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " Oct 30 19:29:57.767371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.767354 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\") pod \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\" (UID: \"df37ff8a-875d-4405-9fe6-b2bfb757d6cb\") " Oct 30 19:29:57.767738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.767711 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca" pod "df37ff8a-875d-4405-9fe6-b2bfb757d6cb" (UID: "df37ff8a-875d-4405-9fe6-b2bfb757d6cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:57.767981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.767961 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.769684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.769656 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca" pod "df37ff8a-875d-4405-9fe6-b2bfb757d6cb" (UID: "df37ff8a-875d-4405-9fe6-b2bfb757d6cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:57.770176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.770142 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp" pod "df37ff8a-875d-4405-9fe6-b2bfb757d6cb" (UID: "df37ff8a-875d-4405-9fe6-b2bfb757d6cb"). InnerVolumeSpecName "kube-api-access-pw4rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:57.869497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.869399 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4rp\" (UniqueName: \"kubernetes.io/projected/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-kube-api-access-pw4rp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:57.869497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:57.869429 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df37ff8a-875d-4405-9fe6-b2bfb757d6cb-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:29:58.070282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.070232 4990 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:58.070477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.070325 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.295337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.295258 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:58.295610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.295586 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:58.296295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.296277 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:29:58.408179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.408085 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" path="/var/lib/kubelet/pods/2b41e2fb-05b2-4784-bd95-02b4094975ce/volumes" Oct 30 19:29:58.486427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.486382 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.565411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.565385 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" Oct 30 19:29:58.565411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.565394 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks" event={"ID":"df37ff8a-875d-4405-9fe6-b2bfb757d6cb","Type":"ContainerDied","Data":"63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b"} Oct 30 19:29:58.565941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.565446 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a7e006752448ff4a5384046fee003dee10b01093575701a30cc27177be8b3b" Oct 30 19:29:58.629431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.629403 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:29:58.630035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630022 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.630079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630037 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.630079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630045 4990 cpu_manager.go:414] "containerMap: removing container" podUID="df37ff8a-875d-4405-9fe6-b2bfb757d6cb" containerName="revoke-keys" Oct 30 19:29:58.630079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630051 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df37ff8a-875d-4405-9fe6-b2bfb757d6cb" containerName="revoke-keys" Oct 30 19:29:58.630079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630063 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.630079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630069 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.630249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630181 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.630249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630189 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="df37ff8a-875d-4405-9fe6-b2bfb757d6cb" containerName="revoke-keys" Oct 30 19:29:58.630249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.630197 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b41e2fb-05b2-4784-bd95-02b4094975ce" containerName="istio-proxy" Oct 30 19:29:58.633208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.633188 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.650364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.650345 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f215f8b0\"" Oct 30 19:29:58.650905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.650889 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f215f8b0-dockercfg-787vr\"" Oct 30 19:29:58.729112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.683061 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:29:58.729112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.685018 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnj9\" (UniqueName: \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.729112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.685076 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.809113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.754854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.809113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.792985 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnj9\" (UniqueName: \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.809113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.793049 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.849804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.837401 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.868111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.855870 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnj9\" (UniqueName: \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\") pod \"payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:58.947712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:58.947681 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:59.067544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.023234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:59.275620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.275582 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:29:59.275869 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:29:59.275845 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bfabb4_1f85_4c4a_91a9_de3558ddfa1b.slice/crio-3e56ca8b9283f8b00a573abe0ea860224161ea2bcb5f3a5c3bc448b12b0fc867 WatchSource:0}: Error finding container 3e56ca8b9283f8b00a573abe0ea860224161ea2bcb5f3a5c3bc448b12b0fc867: Status 404 returned error can't find the container with id 3e56ca8b9283f8b00a573abe0ea860224161ea2bcb5f3a5c3bc448b12b0fc867 Oct 30 19:29:59.579462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.579421 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" event={"ID":"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b","Type":"ContainerStarted","Data":"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b"} Oct 30 19:29:59.580017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.579878 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:29:59.580017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.579918 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" event={"ID":"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b","Type":"ContainerStarted","Data":"3e56ca8b9283f8b00a573abe0ea860224161ea2bcb5f3a5c3bc448b12b0fc867"} Oct 30 19:29:59.615307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:29:59.615251 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" podStartSLOduration=1.615237606 podStartE2EDuration="1.615237606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:59.613619095 +0000 UTC m=+1507.325144463" watchObservedRunningTime="2026-10-30 19:29:59.615237606 +0000 UTC m=+1507.326762971" Oct 30 19:30:00.159934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.159900 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:00.163331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.163310 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.167870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.167844 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:00.171243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.171222 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.175609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.175583 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:00.180240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.180219 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.191867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.191845 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:00.196292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.196271 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:30:00.216950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.216920 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.217150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.217081 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tws5l\" (UniqueName: \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.217333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.217310 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.217480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.217455 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.217574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.217550 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df85r\" (UniqueName: \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.217630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.217606 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8jm\" (UniqueName: \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.234652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.233274 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:00.238405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.238379 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.246276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.243195 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:00.247411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.247391 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.247718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.247700 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:00.254175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.254156 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:00.288186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.288158 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:00.296711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.296677 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:00.319267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319233 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.319439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319292 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df85r\" (UniqueName: \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.319439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319338 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8jm\" (UniqueName: \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.319439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319374 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.319617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319532 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.319617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319564 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgh65\" (UniqueName: \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.319617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319590 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.319785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319669 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tws5l\" (UniqueName: \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.319785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319704 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2bg\" (UniqueName: \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.319910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.319809 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.320034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.320010 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.320544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.320517 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.320631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.320614 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.351313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.351290 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8jm\" (UniqueName: \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.355157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.355132 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df85r\" (UniqueName: \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\") pod \"maas-api-key-cleanup-29889810-vn2ql\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.355264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.355246 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tws5l\" (UniqueName: \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.421455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.421364 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2bg\" (UniqueName: \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.421761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.421735 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.422060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.422035 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgh65\" (UniqueName: \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.422172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.422075 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.422493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.422472 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.422686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.422667 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.441448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.441419 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2bg\" (UniqueName: \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.441817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.441797 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgh65\" (UniqueName: \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:00.481942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.481914 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:00.501379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.501320 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:00.503904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.503873 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:00.527277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.527189 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:00.568121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.567530 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:00.595219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:00.591469 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:01.005236 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:01.005205 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6037ee7_5df5_4aae_944c_aad322ab85e7.slice/crio-d9092ce7520dd96c52a70cf36627134c713413cc849bf678a83c2eec7c902aff WatchSource:0}: Error finding container d9092ce7520dd96c52a70cf36627134c713413cc849bf678a83c2eec7c902aff: Status 404 returned error can't find the container with id d9092ce7520dd96c52a70cf36627134c713413cc849bf678a83c2eec7c902aff Oct 30 19:30:01.008169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.008140 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:01.017161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.017137 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:01.022648 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:01.022624 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf99dfce_79e9_4b9d_a4c9_478902faaf11.slice/crio-e3326446ffbcae00c98c4011614b515c8cc3d1f46efb59769221b883e5db8832 WatchSource:0}: Error finding container e3326446ffbcae00c98c4011614b515c8cc3d1f46efb59769221b883e5db8832: Status 404 returned error can't find the container with id e3326446ffbcae00c98c4011614b515c8cc3d1f46efb59769221b883e5db8832 Oct 30 19:30:01.107755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.107727 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:01.119542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.119520 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:01.150003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.149980 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:01.641001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.640962 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerStarted","Data":"5eaf2a586be9c1f28c7be021cedd8b1d1a36e9006107bd540ca2a5713a3db2d2"} Oct 30 19:30:01.642287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.642261 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" event={"ID":"27ba8e23-1781-4bd1-b691-6b73933c5f5a","Type":"ContainerStarted","Data":"bdadf0ecc92bbdd443c22e91c02041a8d384159820c8645a404216a3cf99c54f"} Oct 30 19:30:01.644012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.643989 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerStarted","Data":"dc8abc0f40a86ef7e8aa6dd979fb77173dc50ce855f8b9f2a57f2b3d90b84ee4"} Oct 30 19:30:01.644152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.644030 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerStarted","Data":"d9092ce7520dd96c52a70cf36627134c713413cc849bf678a83c2eec7c902aff"} Oct 30 19:30:01.645728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.645708 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerStarted","Data":"74e59d980d50458a1456922da536f169e3cd71b772dc9b348f7ebe1fef75114a"} Oct 30 19:30:01.645810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.645744 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerStarted","Data":"e3326446ffbcae00c98c4011614b515c8cc3d1f46efb59769221b883e5db8832"} Oct 30 19:30:01.646979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.646959 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerStarted","Data":"861e265d88bfc44178f95d1c13cceb144b9e02620f71f178487864dfd30d2f9a"} Oct 30 19:30:01.683247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.683173 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" podStartSLOduration=1.6831562899999999 podStartE2EDuration="1.68315629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:01.677443819 +0000 UTC m=+1509.388969185" watchObservedRunningTime="2026-10-30 19:30:01.68315629 +0000 UTC m=+1509.394681656" Oct 30 19:30:01.715724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:01.715658 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" podStartSLOduration=1.715640684 podStartE2EDuration="1.715640684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:01.711318348 +0000 UTC m=+1509.422843715" watchObservedRunningTime="2026-10-30 19:30:01.715640684 +0000 UTC m=+1509.427166050" Oct 30 19:30:02.221492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.221452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:02.659446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.659406 4990 generic.go:356] "Generic (PLEG): container finished" podID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerID="74e59d980d50458a1456922da536f169e3cd71b772dc9b348f7ebe1fef75114a" exitCode=60 Oct 30 19:30:02.659878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.659462 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerDied","Data":"74e59d980d50458a1456922da536f169e3cd71b772dc9b348f7ebe1fef75114a"} Oct 30 19:30:02.659878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.659738 4990 scope.go:122] "RemoveContainer" containerID="74e59d980d50458a1456922da536f169e3cd71b772dc9b348f7ebe1fef75114a" Oct 30 19:30:02.661547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.661525 4990 generic.go:356] "Generic (PLEG): container finished" podID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerID="a184d7cc94a42fcf91db927cfa6e8648ba446d1422ffc5cab9bb8c8a0581f868" exitCode=60 Oct 30 19:30:02.661635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.661600 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerDied","Data":"a184d7cc94a42fcf91db927cfa6e8648ba446d1422ffc5cab9bb8c8a0581f868"} Oct 30 19:30:02.661834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.661813 4990 scope.go:122] "RemoveContainer" containerID="a184d7cc94a42fcf91db927cfa6e8648ba446d1422ffc5cab9bb8c8a0581f868" Oct 30 19:30:02.663685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.663665 4990 generic.go:356] "Generic (PLEG): container finished" podID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerID="5c3b78f7007cc09dbe6a93707072dc05f470ae120bf162561e7c8af75a31785d" exitCode=60 Oct 30 19:30:02.663935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.663910 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerDied","Data":"5c3b78f7007cc09dbe6a93707072dc05f470ae120bf162561e7c8af75a31785d"} Oct 30 19:30:02.664040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.664021 4990 scope.go:122] "RemoveContainer" containerID="5c3b78f7007cc09dbe6a93707072dc05f470ae120bf162561e7c8af75a31785d" Oct 30 19:30:02.665732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.665709 4990 generic.go:356] "Generic (PLEG): container finished" podID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerID="2f2f814b1007de92662aac49ff681f4c2612bf8c3bcc2f84cd9656ed84e8d849" exitCode=60 Oct 30 19:30:02.665834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.665786 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" event={"ID":"27ba8e23-1781-4bd1-b691-6b73933c5f5a","Type":"ContainerDied","Data":"2f2f814b1007de92662aac49ff681f4c2612bf8c3bcc2f84cd9656ed84e8d849"} Oct 30 19:30:02.665989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.665967 4990 scope.go:122] "RemoveContainer" containerID="2f2f814b1007de92662aac49ff681f4c2612bf8c3bcc2f84cd9656ed84e8d849" Oct 30 19:30:02.680838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.680709 4990 generic.go:356] "Generic (PLEG): container finished" podID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerID="dc8abc0f40a86ef7e8aa6dd979fb77173dc50ce855f8b9f2a57f2b3d90b84ee4" exitCode=60 Oct 30 19:30:02.680838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.680817 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerDied","Data":"dc8abc0f40a86ef7e8aa6dd979fb77173dc50ce855f8b9f2a57f2b3d90b84ee4"} Oct 30 19:30:02.681156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:02.681138 4990 scope.go:122] "RemoveContainer" containerID="dc8abc0f40a86ef7e8aa6dd979fb77173dc50ce855f8b9f2a57f2b3d90b84ee4" Oct 30 19:30:03.695281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.695233 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerStarted","Data":"a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649"} Oct 30 19:30:03.697649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.697618 4990 generic.go:356] "Generic (PLEG): container finished" podID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerID="21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352" exitCode=60 Oct 30 19:30:03.697782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.697743 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" event={"ID":"27ba8e23-1781-4bd1-b691-6b73933c5f5a","Type":"ContainerDied","Data":"21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352"} Oct 30 19:30:03.697840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.697803 4990 scope.go:122] "RemoveContainer" containerID="2f2f814b1007de92662aac49ff681f4c2612bf8c3bcc2f84cd9656ed84e8d849" Oct 30 19:30:03.698005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.697990 4990 scope.go:122] "RemoveContainer" containerID="21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352" Oct 30 19:30:03.698342 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:03.698322 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b_odh-ai-gateway-infra(27ba8e23-1781-4bd1-b691-6b73933c5f5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" Oct 30 19:30:03.700871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.700846 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerStarted","Data":"b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49"} Oct 30 19:30:03.703310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.703289 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerStarted","Data":"bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68"} Oct 30 19:30:03.705778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.705755 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerStarted","Data":"c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1"} Oct 30 19:30:03.710500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.710465 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:03.731234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.731191 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" podStartSLOduration=3.731178786 podStartE2EDuration="3.731178786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.729864584 +0000 UTC m=+1511.441389951" watchObservedRunningTime="2026-10-30 19:30:03.731178786 +0000 UTC m=+1511.442704152" Oct 30 19:30:03.844131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:03.844053 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" podStartSLOduration=3.844038611 podStartE2EDuration="3.844038611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.839214382 +0000 UTC m=+1511.550739747" watchObservedRunningTime="2026-10-30 19:30:03.844038611 +0000 UTC m=+1511.555564011" Oct 30 19:30:04.038772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.038739 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.105366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.105260 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:30:04.110856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.110827 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.130412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.130384 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:30:04.135529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.135498 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-87b0dcce-openshift-default-dockercfg-hhjvz\"" Oct 30 19:30:04.184259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184218 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184272 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84xk\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184344 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184413 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184547 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184588 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184676 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184747 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.184823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.184811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.198354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.198296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.286331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.286287 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.287123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.287082 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.288305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.288284 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.288735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.288722 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.290995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.290973 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84xk\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.291470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.291449 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.292150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.292130 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.292668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.292652 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.292802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.292788 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.293022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.292514 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.293138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.288185 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.293243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.286949 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.293566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.293548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.293664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.288620 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.296641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.296616 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.298084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.296877 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.312850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.312775 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.313563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.313532 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84xk\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\") pod \"e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.442462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.442381 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:04.726816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.726731 4990 generic.go:356] "Generic (PLEG): container finished" podID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerID="bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68" exitCode=60 Oct 30 19:30:04.727367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.727233 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerDied","Data":"bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68"} Oct 30 19:30:04.727367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.727294 4990 scope.go:122] "RemoveContainer" containerID="bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68" Oct 30 19:30:04.727978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.727689 4990 scope.go:122] "RemoveContainer" containerID="74e59d980d50458a1456922da536f169e3cd71b772dc9b348f7ebe1fef75114a" Oct 30 19:30:04.727978 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:04.727781 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd_odh-ai-gateway-infra(bf99dfce-79e9-4b9d-a4c9-478902faaf11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" Oct 30 19:30:04.733946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.733915 4990 generic.go:356] "Generic (PLEG): container finished" podID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerID="c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1" exitCode=60 Oct 30 19:30:04.734077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.734059 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerDied","Data":"c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1"} Oct 30 19:30:04.734546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.734523 4990 scope.go:122] "RemoveContainer" containerID="c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1" Oct 30 19:30:04.734952 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:04.734926 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh_odh-ai-gateway-infra(575a2833-f783-4a04-9cd3-15bb47a00a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" Oct 30 19:30:04.737506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.737483 4990 generic.go:356] "Generic (PLEG): container finished" podID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerID="a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649" exitCode=60 Oct 30 19:30:04.737686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.737535 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerDied","Data":"a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649"} Oct 30 19:30:04.737845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.737826 4990 scope.go:122] "RemoveContainer" containerID="a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649" Oct 30 19:30:04.738260 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:04.738239 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-vn2ql_odh-ai-gateway-infra(ec19aa80-c06f-4362-8c09-1eb23751a14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" Oct 30 19:30:04.739883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.739859 4990 scope.go:122] "RemoveContainer" containerID="21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352" Oct 30 19:30:04.740340 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:04.740244 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b_odh-ai-gateway-infra(27ba8e23-1781-4bd1-b691-6b73933c5f5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" Oct 30 19:30:04.742293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.742267 4990 generic.go:356] "Generic (PLEG): container finished" podID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerID="b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49" exitCode=60 Oct 30 19:30:04.742378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.742308 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerDied","Data":"b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49"} Oct 30 19:30:04.742936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.742620 4990 scope.go:122] "RemoveContainer" containerID="b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49" Oct 30 19:30:04.742936 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:04.742895 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz_odh-ai-gateway-infra(e6037ee7-5df5-4aae-944c-aad322ab85e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" Oct 30 19:30:04.746792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.746772 4990 scope.go:122] "RemoveContainer" containerID="a184d7cc94a42fcf91db927cfa6e8648ba446d1422ffc5cab9bb8c8a0581f868" Oct 30 19:30:04.761176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.761155 4990 scope.go:122] "RemoveContainer" containerID="5c3b78f7007cc09dbe6a93707072dc05f470ae120bf162561e7c8af75a31785d" Oct 30 19:30:04.794087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.794067 4990 scope.go:122] "RemoveContainer" containerID="dc8abc0f40a86ef7e8aa6dd979fb77173dc50ce855f8b9f2a57f2b3d90b84ee4" Oct 30 19:30:04.799495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.799463 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:30:04.801837 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:04.801754 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1972b098_4e10_475a_bffd_256073373f8c.slice/crio-9d705598605169c77e1c102a175ff70499f80e265f5cb7d60956713f781e9e6b WatchSource:0}: Error finding container 9d705598605169c77e1c102a175ff70499f80e265f5cb7d60956713f781e9e6b: Status 404 returned error can't find the container with id 9d705598605169c77e1c102a175ff70499f80e265f5cb7d60956713f781e9e6b Oct 30 19:30:04.808929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.808892 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:04.809019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.808973 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:04.809062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.809016 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:04.824392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.824033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.991167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:04.990110 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:05.100419 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:05.100354 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:30:05.753352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.753322 4990 scope.go:122] "RemoveContainer" containerID="b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49" Oct 30 19:30:05.753973 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:05.753671 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz_odh-ai-gateway-infra(e6037ee7-5df5-4aae-944c-aad322ab85e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" Oct 30 19:30:05.755274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.755252 4990 scope.go:122] "RemoveContainer" containerID="bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68" Oct 30 19:30:05.755523 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:05.755501 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd_odh-ai-gateway-infra(bf99dfce-79e9-4b9d-a4c9-478902faaf11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" Oct 30 19:30:05.756850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.756834 4990 scope.go:122] "RemoveContainer" containerID="c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1" Oct 30 19:30:05.757166 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:05.757079 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh_odh-ai-gateway-infra(575a2833-f783-4a04-9cd3-15bb47a00a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" Oct 30 19:30:05.758454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.758436 4990 scope.go:122] "RemoveContainer" containerID="a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649" Oct 30 19:30:05.758710 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:05.758685 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-vn2ql_odh-ai-gateway-infra(ec19aa80-c06f-4362-8c09-1eb23751a14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" Oct 30 19:30:05.759829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.759810 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerStarted","Data":"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9"} Oct 30 19:30:05.759913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.759851 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerStarted","Data":"9d705598605169c77e1c102a175ff70499f80e265f5cb7d60956713f781e9e6b"} Oct 30 19:30:05.832221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.832012 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podStartSLOduration=1.831995509 podStartE2EDuration="1.831995509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.826866756 +0000 UTC m=+1513.538392121" watchObservedRunningTime="2026-10-30 19:30:05.831995509 +0000 UTC m=+1513.543520877" Oct 30 19:30:05.939950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:05.939900 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.443587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.443557 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:06.449627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.449602 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:06.694680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.694569 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.769227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.769189 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:06.770192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.770163 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:30:06.865138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:06.859973 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.029520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:07.029433 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.365207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:07.365171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.176213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:08.176174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.331103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:08.331055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.482372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:08.482270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.635959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:08.635928 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.948890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:08.948855 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:10.088838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:10.088800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:10.574339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:10.574283 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:10.635343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:10.635314 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:30:10.742680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:10.742648 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:10.910056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:10.909965 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:11.438850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:11.438817 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:12.031463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:12.031430 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:12.636423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:12.636388 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:12.782515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:12.782482 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:13.162192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:13.162152 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:14.394321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:14.394281 4990 scope.go:122] "RemoveContainer" containerID="b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49" Oct 30 19:30:14.631871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:14.631837 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:14.848329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:14.848296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:14.891696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:14.891654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerStarted","Data":"9704e13e39c03ff6d2d86d3f6c10d9d1f6c6bb2f581fcd36c13e041e7385ea79"} Oct 30 19:30:15.085352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.085317 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:15.255081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.255037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:15.905153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.905117 4990 generic.go:356] "Generic (PLEG): container finished" podID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerID="9704e13e39c03ff6d2d86d3f6c10d9d1f6c6bb2f581fcd36c13e041e7385ea79" exitCode=60 Oct 30 19:30:15.905611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.905189 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerDied","Data":"9704e13e39c03ff6d2d86d3f6c10d9d1f6c6bb2f581fcd36c13e041e7385ea79"} Oct 30 19:30:15.905611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.905267 4990 scope.go:122] "RemoveContainer" containerID="b8d358bc88929d71f44f8cd7c8463adf9554e79859c9173b3f357f7ce8c8fb49" Oct 30 19:30:15.905789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.905766 4990 scope.go:122] "RemoveContainer" containerID="9704e13e39c03ff6d2d86d3f6c10d9d1f6c6bb2f581fcd36c13e041e7385ea79" Oct 30 19:30:15.906224 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:15.906196 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz_odh-ai-gateway-infra(e6037ee7-5df5-4aae-944c-aad322ab85e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" Oct 30 19:30:15.942001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:15.941979 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:16.394373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.394333 4990 scope.go:122] "RemoveContainer" containerID="c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1" Oct 30 19:30:16.394625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.394604 4990 scope.go:122] "RemoveContainer" containerID="21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352" Oct 30 19:30:16.461159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.461126 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:30:16.469184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.467409 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.494134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.492947 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:30:16.564570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.564541 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-249dd4-openshift-default-dockercfg-gbt9k\"" Oct 30 19:30:16.565000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.564978 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565025 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565161 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4vk\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565202 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565239 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565322 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565352 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565447 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.570264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.565467 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667405 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667461 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667598 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667627 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667779 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667846 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667857 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4vk\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.667982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.667944 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.668174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.668013 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.668653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.668440 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.668653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.668536 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.668831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.668650 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.668831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.668764 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.758167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.755671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.768763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.768730 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.768955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.768737 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.770475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.770449 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.770608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.770508 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4vk\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.803585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.803552 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:16.931211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.931121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerStarted","Data":"ea3725834167ad167f60bac924d3abac3bec0a5b813c9b26e4d3bab5ad9799eb"} Oct 30 19:30:16.945466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:16.945410 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:17.124190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.121046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:17.164116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.163615 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:30:17.174448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.169253 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.222162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.211976 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3bdb4fd2-dockercfg-44bzd\"" Oct 30 19:30:17.222162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.212467 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3bdb4fd2\"" Oct 30 19:30:17.271101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.245187 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8kr\" (UniqueName: \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.271101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.245336 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.285299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.285202 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:30:17.310614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.310247 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:17.315309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.315282 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:30:17.367114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.348519 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8kr\" (UniqueName: \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.367114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.348657 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.367114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.349684 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.426458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.415838 4990 scope.go:122] "RemoveContainer" containerID="a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649" Oct 30 19:30:17.442922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.440517 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8kr\" (UniqueName: \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\") pod \"payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.454712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.454685 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\") pod \"e6037ee7-5df5-4aae-944c-aad322ab85e7\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " Oct 30 19:30:17.455265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.455241 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\" (UniqueName: \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\") pod \"e6037ee7-5df5-4aae-944c-aad322ab85e7\" (UID: \"e6037ee7-5df5-4aae-944c-aad322ab85e7\") " Oct 30 19:30:17.462761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.462723 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:17.464271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.464244 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:17.464454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.464432 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:17.466146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.463901 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca" pod "e6037ee7-5df5-4aae-944c-aad322ab85e7" (UID: "e6037ee7-5df5-4aae-944c-aad322ab85e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:17.500560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.500523 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65" pod "e6037ee7-5df5-4aae-944c-aad322ab85e7" (UID: "e6037ee7-5df5-4aae-944c-aad322ab85e7"). InnerVolumeSpecName "kube-api-access-xgh65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:17.575322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.575272 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6037ee7-5df5-4aae-944c-aad322ab85e7-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:17.575322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.575316 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgh65\" (UniqueName: \"kubernetes.io/projected/e6037ee7-5df5-4aae-944c-aad322ab85e7-kube-api-access-xgh65\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:17.594959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.594925 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:17.919242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.919207 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:30:17.926447 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:17.923399 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e0b7da_3703_4e06_818c_bc21953211da.slice/crio-84e9a49668e8e2a972f393a108776510d160f3ec4aa288e5ccf3a72ace3c0965 WatchSource:0}: Error finding container 84e9a49668e8e2a972f393a108776510d160f3ec4aa288e5ccf3a72ace3c0965: Status 404 returned error can't find the container with id 84e9a49668e8e2a972f393a108776510d160f3ec4aa288e5ccf3a72ace3c0965 Oct 30 19:30:17.952221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.951883 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" event={"ID":"4a8ac718-7380-4a19-882e-ed1d63832f4b","Type":"ContainerStarted","Data":"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82"} Oct 30 19:30:17.952626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.952251 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" event={"ID":"4a8ac718-7380-4a19-882e-ed1d63832f4b","Type":"ContainerStarted","Data":"b634459f4fffa3ee41798fb31c9e71ac8ce45ea4bc58d4638375dff9934ed28e"} Oct 30 19:30:17.954213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.954191 4990 generic.go:356] "Generic (PLEG): container finished" podID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerID="0e6a24e6716b73b0014ec8a8c7d5ea044aeaba6481d97ae44b35e3fd07c4892a" exitCode=60 Oct 30 19:30:17.954316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.954262 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" event={"ID":"27ba8e23-1781-4bd1-b691-6b73933c5f5a","Type":"ContainerDied","Data":"0e6a24e6716b73b0014ec8a8c7d5ea044aeaba6481d97ae44b35e3fd07c4892a"} Oct 30 19:30:17.954385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.954317 4990 scope.go:122] "RemoveContainer" containerID="21acaf2ea20d50361aea29a9e64e7d1cb53a37947226eda89bd3887103b17352" Oct 30 19:30:17.954636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.954619 4990 scope.go:122] "RemoveContainer" containerID="0e6a24e6716b73b0014ec8a8c7d5ea044aeaba6481d97ae44b35e3fd07c4892a" Oct 30 19:30:17.954859 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:17.954835 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b_odh-ai-gateway-infra(27ba8e23-1781-4bd1-b691-6b73933c5f5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" Oct 30 19:30:17.956893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.956557 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" Oct 30 19:30:17.956969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.956950 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz" event={"ID":"e6037ee7-5df5-4aae-944c-aad322ab85e7","Type":"ContainerDied","Data":"d9092ce7520dd96c52a70cf36627134c713413cc849bf678a83c2eec7c902aff"} Oct 30 19:30:17.960378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.960349 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" event={"ID":"e0e0b7da-3703-4e06-818c-bc21953211da","Type":"ContainerStarted","Data":"84e9a49668e8e2a972f393a108776510d160f3ec4aa288e5ccf3a72ace3c0965"} Oct 30 19:30:17.962453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.962431 4990 generic.go:356] "Generic (PLEG): container finished" podID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerID="ea3725834167ad167f60bac924d3abac3bec0a5b813c9b26e4d3bab5ad9799eb" exitCode=60 Oct 30 19:30:17.962614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.962502 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerDied","Data":"ea3725834167ad167f60bac924d3abac3bec0a5b813c9b26e4d3bab5ad9799eb"} Oct 30 19:30:17.962845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.962827 4990 scope.go:122] "RemoveContainer" containerID="ea3725834167ad167f60bac924d3abac3bec0a5b813c9b26e4d3bab5ad9799eb" Oct 30 19:30:17.963077 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:17.963056 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh_odh-ai-gateway-infra(575a2833-f783-4a04-9cd3-15bb47a00a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" Oct 30 19:30:17.978628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.978593 4990 scope.go:122] "RemoveContainer" containerID="9704e13e39c03ff6d2d86d3f6c10d9d1f6c6bb2f581fcd36c13e041e7385ea79" Oct 30 19:30:17.985747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.985724 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:17.997823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:17.997801 4990 scope.go:122] "RemoveContainer" containerID="c5a5bca0462a854852d24ecd41912a08f64d59c0f4e5a3ffce78e3b3162f8ed1" Oct 30 19:30:18.003997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.003948 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podStartSLOduration=2.003933527 podStartE2EDuration="2.003933527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:17.994285792 +0000 UTC m=+1525.705811159" watchObservedRunningTime="2026-10-30 19:30:18.003933527 +0000 UTC m=+1525.715458893" Oct 30 19:30:18.067186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.067128 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:18.072866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.072838 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889810-dwlcz"] Oct 30 19:30:18.394187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.394084 4990 scope.go:122] "RemoveContainer" containerID="bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68" Oct 30 19:30:18.407498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.407471 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" path="/var/lib/kubelet/pods/e6037ee7-5df5-4aae-944c-aad322ab85e7/volumes" Oct 30 19:30:18.804053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.804021 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:18.805299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.805276 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:18.805378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.805328 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:18.855348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.855267 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:18.996759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.996610 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" event={"ID":"e0e0b7da-3703-4e06-818c-bc21953211da","Type":"ContainerStarted","Data":"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe"} Oct 30 19:30:18.997365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:18.997307 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:19.000740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.000712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerStarted","Data":"8c224aaed5f321186db7389d472333ee673267c6edd8e3b98f531d655fc98666"} Oct 30 19:30:19.008865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.007261 4990 generic.go:356] "Generic (PLEG): container finished" podID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerID="02366b630b20085b9aaf88d881134d59336165cd62768c94b37852eeb516c1c1" exitCode=60 Oct 30 19:30:19.008865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.007892 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerDied","Data":"02366b630b20085b9aaf88d881134d59336165cd62768c94b37852eeb516c1c1"} Oct 30 19:30:19.008865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.007929 4990 scope.go:122] "RemoveContainer" containerID="a7676db5b36b5e5db22031aaca9665cb23f529a92ac536ae454056334caf5649" Oct 30 19:30:19.008865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.008247 4990 scope.go:122] "RemoveContainer" containerID="02366b630b20085b9aaf88d881134d59336165cd62768c94b37852eeb516c1c1" Oct 30 19:30:19.008865 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:19.008559 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-vn2ql_odh-ai-gateway-infra(ec19aa80-c06f-4362-8c09-1eb23751a14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" Oct 30 19:30:19.036403 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.036352 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" podStartSLOduration=2.036337441 podStartE2EDuration="2.036337441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:19.03270648 +0000 UTC m=+1526.744231846" watchObservedRunningTime="2026-10-30 19:30:19.036337441 +0000 UTC m=+1526.747862798" Oct 30 19:30:19.050119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.050077 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:19.338831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.338798 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:19.521195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.521165 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\" (UniqueName: \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\") pod \"575a2833-f783-4a04-9cd3-15bb47a00a02\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " Oct 30 19:30:19.521315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.521243 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\") pod \"575a2833-f783-4a04-9cd3-15bb47a00a02\" (UID: \"575a2833-f783-4a04-9cd3-15bb47a00a02\") " Oct 30 19:30:19.521604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.521582 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca" pod "575a2833-f783-4a04-9cd3-15bb47a00a02" (UID: "575a2833-f783-4a04-9cd3-15bb47a00a02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.522002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.521985 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575a2833-f783-4a04-9cd3-15bb47a00a02-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:19.523215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.523193 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm" pod "575a2833-f783-4a04-9cd3-15bb47a00a02" (UID: "575a2833-f783-4a04-9cd3-15bb47a00a02"). InnerVolumeSpecName "kube-api-access-6s8jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.604224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.604188 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:19.623291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.623261 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8jm\" (UniqueName: \"kubernetes.io/projected/575a2833-f783-4a04-9cd3-15bb47a00a02-kube-api-access-6s8jm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:19.725004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.724964 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\") pod \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " Oct 30 19:30:19.725397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.725372 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\" (UniqueName: \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\") pod \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\" (UID: \"27ba8e23-1781-4bd1-b691-6b73933c5f5a\") " Oct 30 19:30:19.725605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.725468 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca" pod "27ba8e23-1781-4bd1-b691-6b73933c5f5a" (UID: "27ba8e23-1781-4bd1-b691-6b73933c5f5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.726520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.726497 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ba8e23-1781-4bd1-b691-6b73933c5f5a-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:19.728012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.727981 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg" pod "27ba8e23-1781-4bd1-b691-6b73933c5f5a" (UID: "27ba8e23-1781-4bd1-b691-6b73933c5f5a"). InnerVolumeSpecName "kube-api-access-vf2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.804593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.804562 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:19.804752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.804614 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:19.828343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:19.828318 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2bg\" (UniqueName: \"kubernetes.io/projected/27ba8e23-1781-4bd1-b691-6b73933c5f5a-kube-api-access-vf2bg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:20.022143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.022032 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" event={"ID":"27ba8e23-1781-4bd1-b691-6b73933c5f5a","Type":"ContainerDied","Data":"bdadf0ecc92bbdd443c22e91c02041a8d384159820c8645a404216a3cf99c54f"} Oct 30 19:30:20.022143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.022125 4990 scope.go:122] "RemoveContainer" containerID="0e6a24e6716b73b0014ec8a8c7d5ea044aeaba6481d97ae44b35e3fd07c4892a" Oct 30 19:30:20.022580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.022242 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b" Oct 30 19:30:20.024890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.024865 4990 generic.go:356] "Generic (PLEG): container finished" podID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerID="8c224aaed5f321186db7389d472333ee673267c6edd8e3b98f531d655fc98666" exitCode=60 Oct 30 19:30:20.025022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.024928 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerDied","Data":"8c224aaed5f321186db7389d472333ee673267c6edd8e3b98f531d655fc98666"} Oct 30 19:30:20.025414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.025395 4990 scope.go:122] "RemoveContainer" containerID="8c224aaed5f321186db7389d472333ee673267c6edd8e3b98f531d655fc98666" Oct 30 19:30:20.025795 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:20.025772 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd_odh-ai-gateway-infra(bf99dfce-79e9-4b9d-a4c9-478902faaf11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" Oct 30 19:30:20.026146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.026129 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" event={"ID":"575a2833-f783-4a04-9cd3-15bb47a00a02","Type":"ContainerDied","Data":"861e265d88bfc44178f95d1c13cceb144b9e02620f71f178487864dfd30d2f9a"} Oct 30 19:30:20.026681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.026664 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh" Oct 30 19:30:20.037128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.037110 4990 scope.go:122] "RemoveContainer" containerID="bf8835c1c869bd577a23b838b93e9ea7e45021faf70c40a703f1318f494c3b68" Oct 30 19:30:20.052630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.052612 4990 scope.go:122] "RemoveContainer" containerID="ea3725834167ad167f60bac924d3abac3bec0a5b813c9b26e4d3bab5ad9799eb" Oct 30 19:30:20.098857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.098826 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:20.140124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.140073 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:20.147490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.147392 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889810-hdt5b"] Oct 30 19:30:20.228497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.220353 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:20.228497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.225565 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:20.235374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.233667 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889810-q4pfh"] Oct 30 19:30:20.332726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.332696 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:20.415228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.415198 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" path="/var/lib/kubelet/pods/27ba8e23-1781-4bd1-b691-6b73933c5f5a/volumes" Oct 30 19:30:20.416176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.416152 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" path="/var/lib/kubelet/pods/575a2833-f783-4a04-9cd3-15bb47a00a02/volumes" Oct 30 19:30:20.442185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.442163 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\" (UniqueName: \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\") pod \"ec19aa80-c06f-4362-8c09-1eb23751a14e\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " Oct 30 19:30:20.442314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.442239 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\") pod \"ec19aa80-c06f-4362-8c09-1eb23751a14e\" (UID: \"ec19aa80-c06f-4362-8c09-1eb23751a14e\") " Oct 30 19:30:20.442609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.442584 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca" pod "ec19aa80-c06f-4362-8c09-1eb23751a14e" (UID: "ec19aa80-c06f-4362-8c09-1eb23751a14e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.442862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.442843 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19aa80-c06f-4362-8c09-1eb23751a14e-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:20.444381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.444356 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r" pod "ec19aa80-c06f-4362-8c09-1eb23751a14e" (UID: "ec19aa80-c06f-4362-8c09-1eb23751a14e"). InnerVolumeSpecName "kube-api-access-df85r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.544509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.544471 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df85r\" (UniqueName: \"kubernetes.io/projected/ec19aa80-c06f-4362-8c09-1eb23751a14e-kube-api-access-df85r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:20.804715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.804679 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:20.804895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.804734 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:20.983107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:20.983050 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.048294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.048271 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" Oct 30 19:30:21.048738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.048302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql" event={"ID":"ec19aa80-c06f-4362-8c09-1eb23751a14e","Type":"ContainerDied","Data":"5eaf2a586be9c1f28c7be021cedd8b1d1a36e9006107bd540ca2a5713a3db2d2"} Oct 30 19:30:21.048738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.048373 4990 scope.go:122] "RemoveContainer" containerID="02366b630b20085b9aaf88d881134d59336165cd62768c94b37852eeb516c1c1" Oct 30 19:30:21.131581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.131107 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:21.137167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.137138 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-vn2ql"] Oct 30 19:30:21.301938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.301912 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:21.356992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.356902 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\" (UniqueName: \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\") pod \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " Oct 30 19:30:21.357227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.357204 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\") pod \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\" (UID: \"bf99dfce-79e9-4b9d-a4c9-478902faaf11\") " Oct 30 19:30:21.357944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.357921 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca" pod "bf99dfce-79e9-4b9d-a4c9-478902faaf11" (UID: "bf99dfce-79e9-4b9d-a4c9-478902faaf11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:21.359194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.359170 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l" pod "bf99dfce-79e9-4b9d-a4c9-478902faaf11" (UID: "bf99dfce-79e9-4b9d-a4c9-478902faaf11"). InnerVolumeSpecName "kube-api-access-tws5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.459552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.459517 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf99dfce-79e9-4b9d-a4c9-478902faaf11-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:21.459724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.459556 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tws5l\" (UniqueName: \"kubernetes.io/projected/bf99dfce-79e9-4b9d-a4c9-478902faaf11-kube-api-access-tws5l\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:21.804584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.804554 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:21.804776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:21.804611 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:22.064491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.064402 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" event={"ID":"bf99dfce-79e9-4b9d-a4c9-478902faaf11","Type":"ContainerDied","Data":"e3326446ffbcae00c98c4011614b515c8cc3d1f46efb59769221b883e5db8832"} Oct 30 19:30:22.064926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.064417 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd" Oct 30 19:30:22.064926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.064498 4990 scope.go:122] "RemoveContainer" containerID="8c224aaed5f321186db7389d472333ee673267c6edd8e3b98f531d655fc98666" Oct 30 19:30:22.138501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.138467 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:22.147726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.147701 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889810-f6dxd"] Oct 30 19:30:22.405001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.404910 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" path="/var/lib/kubelet/pods/bf99dfce-79e9-4b9d-a4c9-478902faaf11/volumes" Oct 30 19:30:22.405474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.405451 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" path="/var/lib/kubelet/pods/ec19aa80-c06f-4362-8c09-1eb23751a14e/volumes" Oct 30 19:30:22.733868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.733770 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.806115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.805590 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:22.806115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:22.805650 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:23.804762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:23.804728 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:30:23.805294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:23.804797 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 19:30:24.541078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:24.537861 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.808541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:24.808512 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:24.808992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:24.808919 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:24.809696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:24.809679 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:30:24.925668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:24.925628 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.404720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:25.404687 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.562115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:25.554554 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:26.381564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:26.381523 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:26.534843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:26.534790 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:28.722754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:28.722716 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.331730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:29.331696 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.490314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:29.490280 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:30.029446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:30.029419 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:30:30.628004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:30.627968 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.091307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.091269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.213125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213064 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 19:30:31.213786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213765 4990 cpu_manager.go:414] "containerMap: removing container" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.213786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213786 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213815 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213823 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213836 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213841 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213851 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213855 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213862 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213868 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213877 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213884 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213892 4990 cpu_manager.go:414] "containerMap: removing container" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213896 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213909 4990 cpu_manager.go:414] "containerMap: removing container" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213914 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213924 4990 cpu_manager.go:414] "containerMap: removing container" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.213929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213930 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213964 4990 cpu_manager.go:414] "containerMap: removing container" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213970 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213980 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.213984 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214104 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214115 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214124 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214130 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214135 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214140 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214148 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214155 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="575a2833-f783-4a04-9cd3-15bb47a00a02" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214164 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214170 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.214726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.214175 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.219792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.219771 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.236224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.236200 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 19:30:31.239767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.239673 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a56cb5\"" Oct 30 19:30:31.239930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.239910 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a56cb5-dockercfg-52rn6\"" Oct 30 19:30:31.253605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.253579 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.369192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.369085 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.369461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.369437 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjn85\" (UniqueName: \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.434630 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435782 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435804 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435820 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435828 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435838 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435847 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435873 4990 cpu_manager.go:414] "containerMap: removing container" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.435881 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.436074 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6037ee7-5df5-4aae-944c-aad322ab85e7" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.436101 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf99dfce-79e9-4b9d-a4c9-478902faaf11" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.436121 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19aa80-c06f-4362-8c09-1eb23751a14e" containerName="cleanup" Oct 30 19:30:31.438379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.436134 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ba8e23-1781-4bd1-b691-6b73933c5f5a" containerName="cleanup" Oct 30 19:30:31.505929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.505883 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.506154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.505989 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn85\" (UniqueName: \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.521795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.508574 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.521795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.519930 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 19:30:31.521795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.520064 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.565787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.565750 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn85\" (UniqueName: \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\") pod \"payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.611672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.611335 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.611960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.611934 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4r7w\" (UniqueName: \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.713918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.713834 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.714120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.713939 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4r7w\" (UniqueName: \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.714628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.714604 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.735678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.735651 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4r7w\" (UniqueName: \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\") pod \"payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:31.834951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.834918 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:31.900112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:31.896864 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:32.165052 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:32.165025 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1564f703_b9e7_4851_8ddd_e1f6b7b0a401.slice/crio-f3887770f68cbfb5538bb56e302017826e5884c55cf52d735a11bf54716addd2 WatchSource:0}: Error finding container f3887770f68cbfb5538bb56e302017826e5884c55cf52d735a11bf54716addd2: Status 404 returned error can't find the container with id f3887770f68cbfb5538bb56e302017826e5884c55cf52d735a11bf54716addd2 Oct 30 19:30:32.165405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:32.165289 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 19:30:32.198668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:32.198643 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" event={"ID":"1564f703-b9e7-4851-8ddd-e1f6b7b0a401","Type":"ContainerStarted","Data":"f3887770f68cbfb5538bb56e302017826e5884c55cf52d735a11bf54716addd2"} Oct 30 19:30:32.257581 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:30:32.257550 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5468dfab_1958_4793_a2e3_f83df3930d12.slice/crio-57a1ecb5fbafc46eccff33b6e8a6a0dd1678245b3285efbe86205f29b976916f WatchSource:0}: Error finding container 57a1ecb5fbafc46eccff33b6e8a6a0dd1678245b3285efbe86205f29b976916f: Status 404 returned error can't find the container with id 57a1ecb5fbafc46eccff33b6e8a6a0dd1678245b3285efbe86205f29b976916f Oct 30 19:30:32.257947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:32.257926 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 19:30:32.724538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:32.724493 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:33.210472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.210418 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" event={"ID":"5468dfab-1958-4793-a2e3-f83df3930d12","Type":"ContainerStarted","Data":"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791"} Oct 30 19:30:33.211011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.210484 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" event={"ID":"5468dfab-1958-4793-a2e3-f83df3930d12","Type":"ContainerStarted","Data":"57a1ecb5fbafc46eccff33b6e8a6a0dd1678245b3285efbe86205f29b976916f"} Oct 30 19:30:33.211209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.211182 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:33.213486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.213456 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" event={"ID":"1564f703-b9e7-4851-8ddd-e1f6b7b0a401","Type":"ContainerStarted","Data":"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856"} Oct 30 19:30:33.213624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.213538 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:33.245212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.245156 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" podStartSLOduration=2.245139557 podStartE2EDuration="2.245139557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:33.243947351 +0000 UTC m=+1540.955472717" watchObservedRunningTime="2026-10-30 19:30:33.245139557 +0000 UTC m=+1540.956664923" Oct 30 19:30:33.284818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:33.284765 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" podStartSLOduration=2.28475165 podStartE2EDuration="2.28475165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:33.27940279 +0000 UTC m=+1540.990928169" watchObservedRunningTime="2026-10-30 19:30:33.28475165 +0000 UTC m=+1540.996277352" Oct 30 19:30:34.694888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:34.694802 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:34.848149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:34.847477 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:34.987657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:34.987565 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:35.098576 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:35.098535 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:30:37.015512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.015475 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.574382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.574350 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw"] Oct 30 19:30:37.579087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.579069 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.591906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.591885 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:37.601491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.601461 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:30:37.609159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.609134 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw"] Oct 30 19:30:37.720904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.720870 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.721077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.720979 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qjq\" (UniqueName: \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.721229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.721215 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.822841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.822804 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qjq\" (UniqueName: \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.823074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.823055 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.823225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.823209 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.823872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.823839 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.825563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.825497 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.839979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.839955 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qjq\" (UniqueName: \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\") pod \"maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:37.894508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:37.894476 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:38.211568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:38.211533 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw"] Oct 30 19:30:38.270969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:38.270943 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" event={"ID":"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28","Type":"ContainerStarted","Data":"867ee7170705ed4a5ec81cd68687015e0b4b7a4e186e1478b189c16ca245c240"} Oct 30 19:30:38.939163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:38.939039 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:39.282207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:39.282121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" event={"ID":"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28","Type":"ContainerStarted","Data":"a29a72099b88e7521e80541a71868d7620b4e3d687dadd13de33c2c04d987a2d"} Oct 30 19:30:39.317216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:39.317158 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" podStartSLOduration=2.317138705 podStartE2EDuration="2.317138705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:39.315336609 +0000 UTC m=+1547.026861975" watchObservedRunningTime="2026-10-30 19:30:39.317138705 +0000 UTC m=+1547.028664069" Oct 30 19:30:40.294574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:40.294541 4990 generic.go:356] "Generic (PLEG): container finished" podID="bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" containerID="a29a72099b88e7521e80541a71868d7620b4e3d687dadd13de33c2c04d987a2d" exitCode=0 Oct 30 19:30:40.295133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:40.294633 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" event={"ID":"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28","Type":"ContainerDied","Data":"a29a72099b88e7521e80541a71868d7620b4e3d687dadd13de33c2c04d987a2d"} Oct 30 19:30:40.737583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:40.737541 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.093241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.093203 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.270074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.270033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.500701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.500674 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:41.592714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.592676 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\") pod \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " Oct 30 19:30:41.592904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.592813 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\") pod \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " Oct 30 19:30:41.592969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.592933 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\" (UniqueName: \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\") pod \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\" (UID: \"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28\") " Oct 30 19:30:41.594727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.594698 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca" pod "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" (UID: "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:41.596595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.596566 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca" pod "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" (UID: "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:41.597226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.597198 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq" pod "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" (UID: "bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28"). InnerVolumeSpecName "kube-api-access-v7qjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:41.695508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.695429 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:41.695508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.695458 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qjq\" (UniqueName: \"kubernetes.io/projected/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-kube-api-access-v7qjq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:41.695508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.695468 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:41.990818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:41.990703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:42.314659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:42.314630 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" Oct 30 19:30:42.314870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:42.314672 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw" event={"ID":"bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28","Type":"ContainerDied","Data":"867ee7170705ed4a5ec81cd68687015e0b4b7a4e186e1478b189c16ca245c240"} Oct 30 19:30:42.314870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:42.314762 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867ee7170705ed4a5ec81cd68687015e0b4b7a4e186e1478b189c16ca245c240" Oct 30 19:30:42.437336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:42.437301 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:44.228689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:44.228663 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 19:30:44.234454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:44.234423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:45.237401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:45.237371 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 19:30:45.690637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:45.690595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:45.855311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:45.855275 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:46.008154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:46.008039 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:47.033103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:47.033067 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p"] Oct 30 19:30:47.042111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:47.042073 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d211bdf-fdx6p"] Oct 30 19:30:47.859641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:47.859600 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:48.339323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:48.339287 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:30:48.412449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:48.412410 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa843b4-abd5-48e0-af05-5d55747e6322" path="/var/lib/kubelet/pods/3fa843b4-abd5-48e0-af05-5d55747e6322/volumes" Oct 30 19:30:48.503634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:48.503591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:48.665876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:48.665791 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.718953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:50.718915 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:50.863718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:50.863674 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.145603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.145566 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.339642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.339594 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" podUID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" containerName="payload-pre-processing" containerID="cri-o://23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b" gracePeriod=25 Oct 30 19:30:53.723975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.723907 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:30:53.764909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.764882 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\") pod \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " Oct 30 19:30:53.766120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.765205 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\" (UniqueName: \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\") pod \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\" (UID: \"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b\") " Oct 30 19:30:53.775604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.775567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume" pod "76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" (UID: "76bfabb4-1f85-4c4a-91a9-de3558ddfa1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:53.806353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.806310 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9" pod "76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" (UID: "76bfabb4-1f85-4c4a-91a9-de3558ddfa1b"). InnerVolumeSpecName "kube-api-access-8jnj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:53.889112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.879409 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jnj9\" (UniqueName: \"kubernetes.io/projected/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-kube-api-access-8jnj9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:53.889112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:53.879447 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:30:54.458593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.458559 4990 generic.go:356] "Generic (PLEG): container finished" podID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" containerID="23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b" exitCode=0 Oct 30 19:30:54.458999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.458614 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" Oct 30 19:30:54.458999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.458640 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" event={"ID":"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b","Type":"ContainerDied","Data":"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b"} Oct 30 19:30:54.458999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.458694 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw" event={"ID":"76bfabb4-1f85-4c4a-91a9-de3558ddfa1b","Type":"ContainerDied","Data":"3e56ca8b9283f8b00a573abe0ea860224161ea2bcb5f3a5c3bc448b12b0fc867"} Oct 30 19:30:54.458999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.458719 4990 scope.go:122] "RemoveContainer" containerID="23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b" Oct 30 19:30:54.503159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.502354 4990 scope.go:122] "RemoveContainer" containerID="23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b" Oct 30 19:30:54.503159 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:30:54.502721 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b\": container with ID starting with 23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b not found: ID does not exist" containerID="23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b" Oct 30 19:30:54.503159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.502755 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b"} err="failed to get container status \"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b\": rpc error: code = NotFound desc = could not find container \"23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b\": container with ID starting with 23dd1e399b7dc8a6868364225a2271153e7d16155968a94eb90019474b2b200b not found: ID does not exist" Oct 30 19:30:54.522940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.522911 4990 scope.go:122] "RemoveContainer" containerID="24111ebde3482af841a2c000e47a34a2708d5b4f52b474ec63d84d79479aed03" Oct 30 19:30:54.524203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.524183 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:30:54.531993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.531972 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f215f8b0-89c7bd7d7-k24sw"] Oct 30 19:30:54.533774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.533753 4990 scope.go:122] "RemoveContainer" containerID="69acea9fc9d6af41e16802a006cd33aa2a14b6740bf539ad5716c93409a7c7e7" Oct 30 19:30:54.543789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.543770 4990 scope.go:122] "RemoveContainer" containerID="d463b78bba4e76b3356e15a54ff3691b8a4a0e492456ff2d652267c5aea4af65" Oct 30 19:30:54.555564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.555543 4990 scope.go:122] "RemoveContainer" containerID="843c653642bab2597a4768395ed222471215d0a9366ef039c66a95941b5e6491" Oct 30 19:30:54.944123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:54.943934 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:56.405643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:56.405598 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" path="/var/lib/kubelet/pods/76bfabb4-1f85-4c4a-91a9-de3558ddfa1b/volumes" Oct 30 19:30:56.582439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:56.582402 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:56.742619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:56.742534 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.158711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.158677 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.309670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.309634 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw"] Oct 30 19:30:57.317154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.317086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.318227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.318204 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f215f8b0-mp7lw"] Oct 30 19:30:57.473334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.473250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:57.699179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:57.699062 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.045851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:58.045814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.406470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:58.406430 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" path="/var/lib/kubelet/pods/bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28/volumes" Oct 30 19:30:58.554539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:58.554501 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:58.831605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:58.831571 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:59.053524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.053492 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:59.532779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.532685 4990 generic.go:356] "Generic (PLEG): container finished" podID="1972b098-4e10-475a-bffd-256073373f8c" containerID="cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9" exitCode=0 Oct 30 19:30:59.532779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.532746 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerDied","Data":"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9"} Oct 30 19:30:59.533436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.533405 4990 scope.go:122] "RemoveContainer" containerID="cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9" Oct 30 19:30:59.535404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.535372 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:59.535505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.535447 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:59.535505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.535491 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:30:59.851198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:30:59.851159 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:00.029115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:00.028684 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:00.546924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:00.546886 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerStarted","Data":"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961"} Oct 30 19:31:01.443121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:01.443076 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:31:01.443630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:01.443297 4990 patch_prober.go:29] interesting pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:31:01.443630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:01.443360 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 19:31:01.825569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:01.825530 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:02.003663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.003621 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27"] Oct 30 19:31:02.004568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004537 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" containerName="revoke-keys" Oct 30 19:31:02.004568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004569 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" containerName="revoke-keys" Oct 30 19:31:02.004757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004612 4990 cpu_manager.go:414] "containerMap: removing container" podUID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" containerName="payload-pre-processing" Oct 30 19:31:02.004757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004621 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" containerName="payload-pre-processing" Oct 30 19:31:02.004933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004915 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd7f47e-5cf8-437b-b7c5-d3c69eb41a28" containerName="revoke-keys" Oct 30 19:31:02.004988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.004941 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bfabb4-1f85-4c4a-91a9-de3558ddfa1b" containerName="payload-pre-processing" Oct 30 19:31:02.008104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.008073 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.022593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.022563 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27"] Oct 30 19:31:02.022858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.022572 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:02.034591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.034555 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:31:02.141788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.141702 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkk6k\" (UniqueName: \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.141957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.141822 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.142044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.142021 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.243304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.243269 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkk6k\" (UniqueName: \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.243503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.243351 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.243573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.243503 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.244373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.244347 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.245924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.245897 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.263711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.263687 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkk6k\" (UniqueName: \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\") pod \"maas-api-revoke-sub-266bd7e1-6kp27\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.341908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.341876 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:02.443275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.443239 4990 patch_prober.go:29] interesting pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:31:02.443701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.443316 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 19:31:02.696879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:02.683262 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27"] Oct 30 19:31:02.704452 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:31:02.702047 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58ef018_0808_49b2_bc82_b830cae35c38.slice/crio-55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994 WatchSource:0}: Error finding container 55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994: Status 404 returned error can't find the container with id 55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994 Oct 30 19:31:03.444015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.443929 4990 patch_prober.go:29] interesting pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:31:03.444015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.443982 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 19:31:03.494193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.494150 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:03.607311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.601461 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" event={"ID":"f58ef018-0808-49b2-bc82-b830cae35c38","Type":"ContainerStarted","Data":"a932127a66cbada3a35994f88201e99fc28a733c7296ee1956446a480d579417"} Oct 30 19:31:03.607311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.601536 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" event={"ID":"f58ef018-0808-49b2-bc82-b830cae35c38","Type":"ContainerStarted","Data":"55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994"} Oct 30 19:31:03.708925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.708810 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" podStartSLOduration=2.708794726 podStartE2EDuration="2.708794726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.689530367 +0000 UTC m=+1571.401055733" watchObservedRunningTime="2026-10-30 19:31:03.708794726 +0000 UTC m=+1571.420320091" Oct 30 19:31:03.722771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:03.722695 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.443763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:04.443731 4990 patch_prober.go:29] interesting pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:31:04.443975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:04.443787 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 19:31:04.611596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:04.611557 4990 generic.go:356] "Generic (PLEG): container finished" podID="f58ef018-0808-49b2-bc82-b830cae35c38" containerID="a932127a66cbada3a35994f88201e99fc28a733c7296ee1956446a480d579417" exitCode=0 Oct 30 19:31:04.612118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:04.611647 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" event={"ID":"f58ef018-0808-49b2-bc82-b830cae35c38","Type":"ContainerDied","Data":"a932127a66cbada3a35994f88201e99fc28a733c7296ee1956446a480d579417"} Oct 30 19:31:05.030769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.030736 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p"] Oct 30 19:31:05.035899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.035864 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252cee94-6455p"] Oct 30 19:31:05.119506 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:31:05.119468 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:31:05.442910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.442882 4990 patch_prober.go:29] interesting pod/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:31:05.443079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.442927 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 19:31:05.620780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.620709 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:05.876419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:05.876392 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:06.032480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.032395 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\") pod \"f58ef018-0808-49b2-bc82-b830cae35c38\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " Oct 30 19:31:06.032644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.032569 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\" (UniqueName: \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\") pod \"f58ef018-0808-49b2-bc82-b830cae35c38\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " Oct 30 19:31:06.032644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.032630 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\") pod \"f58ef018-0808-49b2-bc82-b830cae35c38\" (UID: \"f58ef018-0808-49b2-bc82-b830cae35c38\") " Oct 30 19:31:06.032819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.032790 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca" pod "f58ef018-0808-49b2-bc82-b830cae35c38" (UID: "f58ef018-0808-49b2-bc82-b830cae35c38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:06.033325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.033306 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:06.035216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.035193 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca" pod "f58ef018-0808-49b2-bc82-b830cae35c38" (UID: "f58ef018-0808-49b2-bc82-b830cae35c38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:06.035325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.035243 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k" pod "f58ef018-0808-49b2-bc82-b830cae35c38" (UID: "f58ef018-0808-49b2-bc82-b830cae35c38"). InnerVolumeSpecName "kube-api-access-pkk6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:06.135023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.134987 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkk6k\" (UniqueName: \"kubernetes.io/projected/f58ef018-0808-49b2-bc82-b830cae35c38-kube-api-access-pkk6k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:06.135023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.135026 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ef018-0808-49b2-bc82-b830cae35c38-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:06.402724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.402681 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cec7ba-2d6e-43ee-a612-857a76f8c08a" path="/var/lib/kubelet/pods/42cec7ba-2d6e-43ee-a612-857a76f8c08a/volumes" Oct 30 19:31:06.448326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.448298 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:31:06.448734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.448712 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:31:06.449586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.449563 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:31:06.633890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.633863 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" Oct 30 19:31:06.634381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.633892 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27" event={"ID":"f58ef018-0808-49b2-bc82-b830cae35c38","Type":"ContainerDied","Data":"55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994"} Oct 30 19:31:06.634381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:06.633937 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e0f22cedf4d6ac4f65953ed4c3f7c075a48bb29f7b2db2c15a6a335d445994" Oct 30 19:31:07.436690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:07.434985 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:09.227107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:09.227050 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:10.325509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.325476 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj"] Oct 30 19:31:10.326152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.326138 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f58ef018-0808-49b2-bc82-b830cae35c38" containerName="revoke-keys" Oct 30 19:31:10.326205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.326153 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ef018-0808-49b2-bc82-b830cae35c38" containerName="revoke-keys" Oct 30 19:31:10.326276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.326267 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ef018-0808-49b2-bc82-b830cae35c38" containerName="revoke-keys" Oct 30 19:31:10.329154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.329138 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.349177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.349152 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj"] Oct 30 19:31:10.368476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.368453 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:10.372382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.372360 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:31:10.403350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.403322 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.403537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.403422 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t8d\" (UniqueName: \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.403537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.403463 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.505500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.505462 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t8d\" (UniqueName: \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.505708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.505526 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.505842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.505818 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.506449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.506414 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.508301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.508279 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.524250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.524225 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t8d\" (UniqueName: \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\") pod \"maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.642211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.642144 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:10.957821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:10.957789 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj"] Oct 30 19:31:10.958442 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:31:10.958416 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0364449d_bc91_47a2_ba8a_d89657b9fb47.slice/crio-757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f WatchSource:0}: Error finding container 757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f: Status 404 returned error can't find the container with id 757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f Oct 30 19:31:11.004143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:11.000164 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:11.702918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:11.702880 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" event={"ID":"0364449d-bc91-47a2-ba8a-d89657b9fb47","Type":"ContainerStarted","Data":"07ab362c8cf996facb28f03cf059e5e3a039fbe86a54337eb79e9438ac8a5168"} Oct 30 19:31:11.703334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:11.702947 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" event={"ID":"0364449d-bc91-47a2-ba8a-d89657b9fb47","Type":"ContainerStarted","Data":"757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f"} Oct 30 19:31:11.738228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:11.738177 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" podStartSLOduration=1.738163862 podStartE2EDuration="1.738163862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:11.737145472 +0000 UTC m=+1579.448670837" watchObservedRunningTime="2026-10-30 19:31:11.738163862 +0000 UTC m=+1579.449689228" Oct 30 19:31:12.475755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:12.475711 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:12.723281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:12.723246 4990 generic.go:356] "Generic (PLEG): container finished" podID="0364449d-bc91-47a2-ba8a-d89657b9fb47" containerID="07ab362c8cf996facb28f03cf059e5e3a039fbe86a54337eb79e9438ac8a5168" exitCode=0 Oct 30 19:31:12.723739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:12.723319 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" event={"ID":"0364449d-bc91-47a2-ba8a-d89657b9fb47","Type":"ContainerDied","Data":"07ab362c8cf996facb28f03cf059e5e3a039fbe86a54337eb79e9438ac8a5168"} Oct 30 19:31:13.059004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:13.058953 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:13.809411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:13.809371 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:14.020396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.020360 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:14.080145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.076695 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\") pod \"0364449d-bc91-47a2-ba8a-d89657b9fb47\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " Oct 30 19:31:14.080145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.076984 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\") pod \"0364449d-bc91-47a2-ba8a-d89657b9fb47\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " Oct 30 19:31:14.080145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.077073 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\" (UniqueName: \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\") pod \"0364449d-bc91-47a2-ba8a-d89657b9fb47\" (UID: \"0364449d-bc91-47a2-ba8a-d89657b9fb47\") " Oct 30 19:31:14.080145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.078794 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca" pod "0364449d-bc91-47a2-ba8a-d89657b9fb47" (UID: "0364449d-bc91-47a2-ba8a-d89657b9fb47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:14.109500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.109466 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d" pod "0364449d-bc91-47a2-ba8a-d89657b9fb47" (UID: "0364449d-bc91-47a2-ba8a-d89657b9fb47"). InnerVolumeSpecName "kube-api-access-h7t8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:14.110205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.110180 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca" pod "0364449d-bc91-47a2-ba8a-d89657b9fb47" (UID: "0364449d-bc91-47a2-ba8a-d89657b9fb47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:14.269066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.268972 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:14.269489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.269463 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7t8d\" (UniqueName: \"kubernetes.io/projected/0364449d-bc91-47a2-ba8a-d89657b9fb47-kube-api-access-h7t8d\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:14.269657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.269642 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0364449d-bc91-47a2-ba8a-d89657b9fb47-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:14.685210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.685175 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:14.747617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.747577 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" Oct 30 19:31:14.747617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.747605 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj" event={"ID":"0364449d-bc91-47a2-ba8a-d89657b9fb47","Type":"ContainerDied","Data":"757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f"} Oct 30 19:31:14.747842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.747659 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757a8738e7abac85802ecd735fe16c07043a65c5313b159acace0d3d0efba58f" Oct 30 19:31:14.869115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:14.858325 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.349396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:15.349368 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.527580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:15.527548 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:17.815114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:17.811770 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.135168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:18.134141 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts"] Oct 30 19:31:18.164373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:18.164234 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d94e164-tc9ts"] Oct 30 19:31:18.455602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:18.455462 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.459676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:18.459643 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872f7286-a78a-4cda-8ff9-45a17df32e95" path="/var/lib/kubelet/pods/872f7286-a78a-4cda-8ff9-45a17df32e95/volumes" Oct 30 19:31:20.053422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:20.052584 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.252955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:20.252923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.408977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:20.408845 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.776336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:20.776246 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.908104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:20.908048 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:31:21.104048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:21.104012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:21.261015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:21.260976 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:21.751312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:21.751273 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.078081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:22.078043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.224871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:22.224837 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.809124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:23.806741 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.617390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:24.617355 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.619925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:25.619884 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.909185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:25.909073 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" podUID="e0e0b7da-3703-4e06-818c-bc21953211da" containerName="payload-processing" containerID="cri-o://71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe" gracePeriod=25 Oct 30 19:31:26.213639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.213613 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:31:26.229371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.229331 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.387281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.387230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.393377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.393350 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\" (UniqueName: \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\") pod \"e0e0b7da-3703-4e06-818c-bc21953211da\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " Oct 30 19:31:26.393589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.393565 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\") pod \"e0e0b7da-3703-4e06-818c-bc21953211da\" (UID: \"e0e0b7da-3703-4e06-818c-bc21953211da\") " Oct 30 19:31:26.393897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.393868 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume" pod "e0e0b7da-3703-4e06-818c-bc21953211da" (UID: "e0e0b7da-3703-4e06-818c-bc21953211da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:26.394379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.394355 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e0b7da-3703-4e06-818c-bc21953211da-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:26.395696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.395664 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr" pod "e0e0b7da-3703-4e06-818c-bc21953211da" (UID: "e0e0b7da-3703-4e06-818c-bc21953211da"). InnerVolumeSpecName "kube-api-access-vc8kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:26.495964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.495895 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8kr\" (UniqueName: \"kubernetes.io/projected/e0e0b7da-3703-4e06-818c-bc21953211da-kube-api-access-vc8kr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:26.964170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.964137 4990 generic.go:356] "Generic (PLEG): container finished" podID="e0e0b7da-3703-4e06-818c-bc21953211da" containerID="71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe" exitCode=0 Oct 30 19:31:26.964608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.964263 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" Oct 30 19:31:26.964608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.964268 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" event={"ID":"e0e0b7da-3703-4e06-818c-bc21953211da","Type":"ContainerDied","Data":"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe"} Oct 30 19:31:26.964608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.964324 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw" event={"ID":"e0e0b7da-3703-4e06-818c-bc21953211da","Type":"ContainerDied","Data":"84e9a49668e8e2a972f393a108776510d160f3ec4aa288e5ccf3a72ace3c0965"} Oct 30 19:31:26.964608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.964345 4990 scope.go:122] "RemoveContainer" containerID="71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe" Oct 30 19:31:26.975197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.975179 4990 scope.go:122] "RemoveContainer" containerID="71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe" Oct 30 19:31:26.975461 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:31:26.975441 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe\": container with ID starting with 71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe not found: ID does not exist" containerID="71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe" Oct 30 19:31:26.975533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:26.975472 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe"} err="failed to get container status \"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe\": rpc error: code = NotFound desc = could not find container \"71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe\": container with ID starting with 71224f3fe2245f9934cc8bc41150ad6d16dee1b50cf5d372071f37ac18cf0efe not found: ID does not exist" Oct 30 19:31:27.011778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:27.011750 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:31:27.021456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:27.021430 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3bdb4fd2-69d5c97b6b-dtnxw"] Oct 30 19:31:27.624102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:27.624043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.134544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.134513 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.268924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.268886 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 19:31:28.269842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.269825 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0364449d-bc91-47a2-ba8a-d89657b9fb47" containerName="revoke-keys" Oct 30 19:31:28.269913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.269845 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0364449d-bc91-47a2-ba8a-d89657b9fb47" containerName="revoke-keys" Oct 30 19:31:28.269913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.269878 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e0e0b7da-3703-4e06-818c-bc21953211da" containerName="payload-processing" Oct 30 19:31:28.269913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.269886 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e0b7da-3703-4e06-818c-bc21953211da" containerName="payload-processing" Oct 30 19:31:28.270053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.270041 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e0b7da-3703-4e06-818c-bc21953211da" containerName="payload-processing" Oct 30 19:31:28.270101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.270064 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0364449d-bc91-47a2-ba8a-d89657b9fb47" containerName="revoke-keys" Oct 30 19:31:28.275526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.275505 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.290493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.290468 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 19:31:28.294423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.294401 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-f83282\"" Oct 30 19:31:28.294598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.294577 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-f83282-dockercfg-bkmjm\"" Oct 30 19:31:28.295039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.295016 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.330227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.330147 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.331053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.330621 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh9j\" (UniqueName: \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.419936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.419859 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e0b7da-3703-4e06-818c-bc21953211da" path="/var/lib/kubelet/pods/e0e0b7da-3703-4e06-818c-bc21953211da/volumes" Oct 30 19:31:28.432378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.432059 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh9j\" (UniqueName: \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.434054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.432563 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.435397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.435377 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.439176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.439155 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 19:31:28.481674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.481592 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh9j\" (UniqueName: \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\") pod \"payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.565158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.565086 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 19:31:28.565535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.565472 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.590940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.590908 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:28.636500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.636463 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.636762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.636736 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqkq\" (UniqueName: \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.739631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.738963 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.739631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.739212 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqkq\" (UniqueName: \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.743397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.743368 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.757738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.757711 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqkq\" (UniqueName: \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\") pod \"payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.887249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.887217 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:28.944000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.943965 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 19:31:28.997648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:28.997614 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" event={"ID":"8ae924eb-1d67-4a66-86d5-a09626b1aeaa","Type":"ContainerStarted","Data":"e379693dc7e41b7ddf5c649afb5a935bd8198aa4165eb4c3de35e277d291ac7f"} Oct 30 19:31:29.212251 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:31:29.212215 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b5317d_dff4_4cd2_8f08_ce5b5066cac3.slice/crio-a13ec9bf4e2f4652a154b2ac2132a9676664718a94f3847795211f27dba60b7b WatchSource:0}: Error finding container a13ec9bf4e2f4652a154b2ac2132a9676664718a94f3847795211f27dba60b7b: Status 404 returned error can't find the container with id a13ec9bf4e2f4652a154b2ac2132a9676664718a94f3847795211f27dba60b7b Oct 30 19:31:29.213415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:29.213387 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 19:31:29.531300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:29.531063 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:30.009931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.009888 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" event={"ID":"8ae924eb-1d67-4a66-86d5-a09626b1aeaa","Type":"ContainerStarted","Data":"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949"} Oct 30 19:31:30.010173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.009991 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:30.011822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.011796 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" event={"ID":"54b5317d-dff4-4cd2-8f08-ce5b5066cac3","Type":"ContainerStarted","Data":"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740"} Oct 30 19:31:30.011972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.011840 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" event={"ID":"54b5317d-dff4-4cd2-8f08-ce5b5066cac3","Type":"ContainerStarted","Data":"a13ec9bf4e2f4652a154b2ac2132a9676664718a94f3847795211f27dba60b7b"} Oct 30 19:31:30.011972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.011884 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:30.038429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.038406 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7"] Oct 30 19:31:30.043542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.043519 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a30ea50-9jkr7"] Oct 30 19:31:30.048398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.048360 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" podStartSLOduration=2.04835004 podStartE2EDuration="2.04835004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:30.044193737 +0000 UTC m=+1597.755719102" watchObservedRunningTime="2026-10-30 19:31:30.04835004 +0000 UTC m=+1597.759875406" Oct 30 19:31:30.079170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.079119 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" podStartSLOduration=2.079104871 podStartE2EDuration="2.079104871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:30.076977765 +0000 UTC m=+1597.788503131" watchObservedRunningTime="2026-10-30 19:31:30.079104871 +0000 UTC m=+1597.790630240" Oct 30 19:31:30.403429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:30.403391 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ed9832-f818-49c0-b93a-f878c86421e5" path="/var/lib/kubelet/pods/22ed9832-f818-49c0-b93a-f878c86421e5/volumes" Oct 30 19:31:31.173009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:31.172974 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:33.029297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:33.029222 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:34.605286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:34.605251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:35.098911 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:31:35.098869 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:31:36.288517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:36.288474 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:37.953002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:37.952964 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:39.450749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:39.450712 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.029455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.029424 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h"] Oct 30 19:31:40.039641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.039604 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda86ab9-q4f2h"] Oct 30 19:31:40.214879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.214849 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.429052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.429008 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0575d3bf-f0a3-4673-94b4-787a8d7c282f" path="/var/lib/kubelet/pods/0575d3bf-f0a3-4673-94b4-787a8d7c282f/volumes" Oct 30 19:31:40.491571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.491531 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.691601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.691517 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj"] Oct 30 19:31:40.700997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.700966 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3bdb4fd2-6l9kj"] Oct 30 19:31:40.785350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.785206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.975172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:40.949624 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:41.026328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:41.026299 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 19:31:41.027128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:41.027107 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 19:31:41.746636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:41.746600 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.334144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:42.319225 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.755163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:42.746799 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.755163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:42.748400 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0364449d-bc91-47a2-ba8a-d89657b9fb47" path="/var/lib/kubelet/pods/0364449d-bc91-47a2-ba8a-d89657b9fb47/volumes" Oct 30 19:31:42.895110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:42.889194 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:42.895110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:42.889260 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:43.727158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:43.727123 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:44.377025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.376987 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4"] Oct 30 19:31:44.381160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.381139 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.395288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.395264 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:44.395607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.395588 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:31:44.412590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.412561 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4"] Oct 30 19:31:44.550829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.550799 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.551285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.551261 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqc9\" (UniqueName: \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.551429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.551404 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.652657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.652569 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqc9\" (UniqueName: \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.652800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.652670 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.652800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.652775 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.653463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.653436 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.655359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.655329 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.671519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.671492 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqc9\" (UniqueName: \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\") pod \"maas-api-revoke-sub-48aa6575-fxzt4\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:44.699186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:44.699162 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:45.032574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.032547 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4"] Oct 30 19:31:45.034013 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:31:45.033981 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adc20a3_8b76_4ea9_88d7_ced3860b52f6.slice/crio-95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36 WatchSource:0}: Error finding container 95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36: Status 404 returned error can't find the container with id 95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36 Oct 30 19:31:45.737801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.737764 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" event={"ID":"1adc20a3-8b76-4ea9-88d7-ced3860b52f6","Type":"ContainerStarted","Data":"1de69bbfcf7cdd6225d9b334da35dc3a98a559d365e8c8034a4766f13f7f399e"} Oct 30 19:31:45.738240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.737818 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" event={"ID":"1adc20a3-8b76-4ea9-88d7-ced3860b52f6","Type":"ContainerStarted","Data":"95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36"} Oct 30 19:31:45.778510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.778450 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" podStartSLOduration=1.7784313790000001 podStartE2EDuration="1.778431379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:45.772949974 +0000 UTC m=+1613.484475340" watchObservedRunningTime="2026-10-30 19:31:45.778431379 +0000 UTC m=+1613.489956749" Oct 30 19:31:45.864185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.864154 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 19:31:45.871640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.870084 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:45.883467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.883416 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-f83282-serving-cert\"" Oct 30 19:31:45.889590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.889565 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 19:31:45.979852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.979820 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdjw\" (UniqueName: \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:45.980041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.979889 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:45.980041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.979968 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:45.980189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.980173 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:45.994691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:45.994662 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:46.081829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.081786 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.082047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.082027 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdjw\" (UniqueName: \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.082223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.082171 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.082223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.082212 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.082525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.082499 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.084852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.084804 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.084951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.084865 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.100066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.100035 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdjw\" (UniqueName: \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\") pod \"maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.134860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.134828 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:46.188475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.188442 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:46.550650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.550595 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 19:31:46.551904 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:31:46.551880 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed5c899_84c4_4eb6_b6e7_9c716822b923.slice/crio-fa1048e4bc8578c2b77dca83261a512f5c36f507c8827b202e2f2f092ffe5a6b WatchSource:0}: Error finding container fa1048e4bc8578c2b77dca83261a512f5c36f507c8827b202e2f2f092ffe5a6b: Status 404 returned error can't find the container with id fa1048e4bc8578c2b77dca83261a512f5c36f507c8827b202e2f2f092ffe5a6b Oct 30 19:31:46.749305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.749269 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" event={"ID":"9ed5c899-84c4-4eb6-b6e7-9c716822b923","Type":"ContainerStarted","Data":"fa1048e4bc8578c2b77dca83261a512f5c36f507c8827b202e2f2f092ffe5a6b"} Oct 30 19:31:46.750993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.750965 4990 generic.go:356] "Generic (PLEG): container finished" podID="1adc20a3-8b76-4ea9-88d7-ced3860b52f6" containerID="1de69bbfcf7cdd6225d9b334da35dc3a98a559d365e8c8034a4766f13f7f399e" exitCode=0 Oct 30 19:31:46.751150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:46.751050 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" event={"ID":"1adc20a3-8b76-4ea9-88d7-ced3860b52f6","Type":"ContainerDied","Data":"1de69bbfcf7cdd6225d9b334da35dc3a98a559d365e8c8034a4766f13f7f399e"} Oct 30 19:31:47.760556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:47.760512 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" event={"ID":"9ed5c899-84c4-4eb6-b6e7-9c716822b923","Type":"ContainerStarted","Data":"a2bffee85c928f30402bd6786def685682a79b255a5e17c831bc590f0d93346e"} Oct 30 19:31:47.761039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:47.760580 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:47.805825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:47.805759 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" podStartSLOduration=2.425619761 podStartE2EDuration="2.805739619s" totalImagesPullingTime="380.119858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:47.799030725 +0000 UTC m=+1615.510556091" watchObservedRunningTime="2026-10-30 19:31:47.805739619 +0000 UTC m=+1615.517264987" Oct 30 19:31:47.916199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:47.916160 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:47.950841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:47.950812 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:48.028160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.026004 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\" (UniqueName: \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\") pod \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " Oct 30 19:31:48.028160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.026062 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\") pod \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " Oct 30 19:31:48.028160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.026311 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\") pod \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\" (UID: \"1adc20a3-8b76-4ea9-88d7-ced3860b52f6\") " Oct 30 19:31:48.028160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.027812 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca" pod "1adc20a3-8b76-4ea9-88d7-ced3860b52f6" (UID: "1adc20a3-8b76-4ea9-88d7-ced3860b52f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:48.053017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.048639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9" pod "1adc20a3-8b76-4ea9-88d7-ced3860b52f6" (UID: "1adc20a3-8b76-4ea9-88d7-ced3860b52f6"). InnerVolumeSpecName "kube-api-access-nqqc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:48.086576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.068370 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca" pod "1adc20a3-8b76-4ea9-88d7-ced3860b52f6" (UID: "1adc20a3-8b76-4ea9-88d7-ced3860b52f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:48.110244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.110204 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:48.136738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.128957 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqqc9\" (UniqueName: \"kubernetes.io/projected/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-kube-api-access-nqqc9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:48.136738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.128988 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:48.136738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.129004 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1adc20a3-8b76-4ea9-88d7-ced3860b52f6-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:31:48.510469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.510429 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:48.770625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.770540 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" Oct 30 19:31:48.770625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.770574 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4" event={"ID":"1adc20a3-8b76-4ea9-88d7-ced3860b52f6","Type":"ContainerDied","Data":"95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36"} Oct 30 19:31:48.770625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:48.770626 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95e33e79b7c8470b06ee83c0b9d01f8025615783347d89a7ab0a679d8452cd36" Oct 30 19:31:49.824313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:49.824270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:50.063297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:50.063263 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:50.866248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:50.866209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:52.028863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:52.028829 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx"] Oct 30 19:31:52.037751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:52.037726 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fcf7fd3-9qkqx"] Oct 30 19:31:52.405150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:52.405114 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430b6da5-3216-4ceb-9e67-f87202ad6dbd" path="/var/lib/kubelet/pods/430b6da5-3216-4ceb-9e67-f87202ad6dbd/volumes" Oct 30 19:31:52.574117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:52.574052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:53.776929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:53.776895 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 19:31:54.439894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:54.439852 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:54.702130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:54.702014 4990 scope.go:122] "RemoveContainer" containerID="2ab49e4042b02fd608a232dbe38a996e49e0ca128a98e354a1fea69e7c5eb0ca" Oct 30 19:31:56.032464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:56.032428 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84"] Oct 30 19:31:56.040055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:56.040019 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.042826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:56.042799 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958634c2-r2n84"] Oct 30 19:31:56.402777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:56.402736 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc0d220-4650-40d9-9342-2fe875bd2e51" path="/var/lib/kubelet/pods/7fc0d220-4650-40d9-9342-2fe875bd2e51/volumes" Oct 30 19:31:57.811221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:57.811187 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.067117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:59.066538 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.859550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:31:59.859517 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.028118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:01.028072 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld"] Oct 30 19:32:01.037699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:01.037673 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fb3b09-64nld"] Oct 30 19:32:01.643494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:01.643460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.403482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:02.403440 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ed2791-9194-4ed7-a4bb-e10434384b7d" path="/var/lib/kubelet/pods/d5ed2791-9194-4ed7-a4bb-e10434384b7d/volumes" Oct 30 19:32:03.450278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:03.450243 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:05.097066 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:05.096981 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:32:05.217658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:05.217620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.022852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.022803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.371944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.371904 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.774753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.774671 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:32:07.775953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.775923 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1adc20a3-8b76-4ea9-88d7-ced3860b52f6" containerName="revoke-keys" Oct 30 19:32:07.775953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.775951 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adc20a3-8b76-4ea9-88d7-ced3860b52f6" containerName="revoke-keys" Oct 30 19:32:07.776214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.776198 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adc20a3-8b76-4ea9-88d7-ced3860b52f6" containerName="revoke-keys" Oct 30 19:32:07.782423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.782400 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.804669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.804643 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ccf8bd61-openshift-default-dockercfg-6xt7q\"" Oct 30 19:32:07.807549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.807425 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:32:07.868084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.868052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:07.876372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876342 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwt4s\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876483 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876600 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876658 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876769 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876833 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.876892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.876865 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.877045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.877030 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.877122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.877060 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.980138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.980077 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.981376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.981352 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.981816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.981785 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.982312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.982286 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.982598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.982562 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.994110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.982746 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.994506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.982428 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.994663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.981242 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.994785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.984425 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.995058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.995041 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.995592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.995572 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.996196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.995991 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.996686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.995506 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:07.996860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:07.996155 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwt4s\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.015445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.004360 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.015445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.004905 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.043585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.043491 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:08.062200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.047987 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.062200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.060025 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwt4s\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\") pod \"e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.108114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.107493 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:08.448084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.448058 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:32:08.450496 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:08.450303 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6ceb2b_642a_499e_ae5a_46d851ba2b57.slice/crio-734edba85becf4151962d64870ea33fb3440e3b9a69038da029de8703460bbba WatchSource:0}: Error finding container 734edba85becf4151962d64870ea33fb3440e3b9a69038da029de8703460bbba: Status 404 returned error can't find the container with id 734edba85becf4151962d64870ea33fb3440e3b9a69038da029de8703460bbba Oct 30 19:32:08.453314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.453275 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:08.453432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.453357 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:08.453432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.453406 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:08.834521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.834463 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 19:32:08.888522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.882404 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 19:32:08.888522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.882539 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:08.903429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.902773 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6b7ccc-serving-cert\"" Oct 30 19:32:08.950962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:08.950925 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.033121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.033075 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h"] Oct 30 19:32:09.039673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.039642 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a586460-dp64h"] Oct 30 19:32:09.044764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.044733 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.045243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.045032 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.045243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.045107 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlxc7\" (UniqueName: \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.045243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.045154 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.085438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.085349 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerStarted","Data":"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7"} Oct 30 19:32:09.085743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.085719 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerStarted","Data":"734edba85becf4151962d64870ea33fb3440e3b9a69038da029de8703460bbba"} Oct 30 19:32:09.095331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.093562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.108390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.108358 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:09.114789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.114759 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:09.132158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.132086 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podStartSLOduration=2.132068382 podStartE2EDuration="2.132068382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:09.127595767 +0000 UTC m=+1636.839121158" watchObservedRunningTime="2026-10-30 19:32:09.132068382 +0000 UTC m=+1636.843593747" Oct 30 19:32:09.146931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.146892 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.147197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.147179 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.147285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.147232 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlxc7\" (UniqueName: \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.147285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.147274 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.148186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.148161 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.148648 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:09.148629 4990 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-serving-cert: secret "maas-api-e2e-shared-a-w2-6b7ccc-serving-cert" not found Oct 30 19:32:09.148737 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:09.148704 4990 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls podName:8b663277-a13d-4f75-a96e-2dde01e07d43 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:09.648684268 +0000 UTC m=+1637.360209618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls") pod "maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" (UID: "8b663277-a13d-4f75-a96e-2dde01e07d43") : secret "maas-api-e2e-shared-a-w2-6b7ccc-serving-cert" not found Oct 30 19:32:09.150731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.150700 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.168906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.168876 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlxc7\" (UniqueName: \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.473736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.473639 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.656466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.656435 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.659006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.658981 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:09.832614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:09.832583 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:10.093841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:10.093765 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:10.094850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:10.094824 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:32:10.162269 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:10.162231 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b663277_a13d_4f75_a96e_2dde01e07d43.slice/crio-a38cc568145aa3a5036a729a58b7154014c86f5ea3d71e4fd55c494fb2649b16 WatchSource:0}: Error finding container a38cc568145aa3a5036a729a58b7154014c86f5ea3d71e4fd55c494fb2649b16: Status 404 returned error can't find the container with id a38cc568145aa3a5036a729a58b7154014c86f5ea3d71e4fd55c494fb2649b16 Oct 30 19:32:10.173148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:10.168758 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 19:32:10.416774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:10.416672 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58d5150-961e-4b0d-9468-adc3633c8e36" path="/var/lib/kubelet/pods/b58d5150-961e-4b0d-9468-adc3633c8e36/volumes" Oct 30 19:32:11.103586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.103541 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" event={"ID":"8b663277-a13d-4f75-a96e-2dde01e07d43","Type":"ContainerStarted","Data":"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129"} Oct 30 19:32:11.104069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.103614 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" event={"ID":"8b663277-a13d-4f75-a96e-2dde01e07d43","Type":"ContainerStarted","Data":"a38cc568145aa3a5036a729a58b7154014c86f5ea3d71e4fd55c494fb2649b16"} Oct 30 19:32:11.104069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.104006 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:11.142164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.142109 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" podStartSLOduration=2.791885546 podStartE2EDuration="3.142082294s" totalImagesPullingTime="350.196748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:11.137252948 +0000 UTC m=+1638.848778315" watchObservedRunningTime="2026-10-30 19:32:11.142082294 +0000 UTC m=+1638.853607660" Oct 30 19:32:11.324243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.324210 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.763519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.763483 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:11.980485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:11.980448 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:12.210028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:12.209994 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:13.032050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.032018 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h"] Oct 30 19:32:13.048265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.048233 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ede2e7-tbq5h"] Oct 30 19:32:13.105101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.105071 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9238-7fac8c"] Oct 30 19:32:13.109808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.109792 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:13.115231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.115205 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xt6p\" (UniqueName: \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\") pod \"test-curl-9238-7fac8c\" (UID: \"11ed0526-2e95-4f11-a73d-03970c26a29f\") " pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:13.122878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.122856 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:32:13.128992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.128965 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-7fac8c"] Oct 30 19:32:13.217078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.217045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xt6p\" (UniqueName: \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\") pod \"test-curl-9238-7fac8c\" (UID: \"11ed0526-2e95-4f11-a73d-03970c26a29f\") " pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:13.235517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.235488 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xt6p\" (UniqueName: \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\") pod \"test-curl-9238-7fac8c\" (UID: \"11ed0526-2e95-4f11-a73d-03970c26a29f\") " pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:13.426631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.426602 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:13.546220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.546064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:13.752829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.752789 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-7fac8c"] Oct 30 19:32:13.756658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.756604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:13.782079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.782028 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:32:13.794179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.794151 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:13.845142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.820595 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ccf8bd61\"" Oct 30 19:32:13.845142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.820833 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ccf8bd61-dockercfg-96fv9\"" Oct 30 19:32:13.845142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.828166 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:13.845142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.828384 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f27\" (UniqueName: \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:13.875510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.867909 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:32:14.028600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.937379 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:14.028600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.937629 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f27\" (UniqueName: \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:14.028600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:13.951189 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:14.085217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.085152 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f27\" (UniqueName: \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:14.138813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.138783 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-7fac8c" event={"ID":"11ed0526-2e95-4f11-a73d-03970c26a29f","Type":"ContainerStarted","Data":"acf02b6d16cd474b60e52a663ad5dd472f1e1dc14bbb99bdbe8172c1c9abfee4"} Oct 30 19:32:14.177022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.176972 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9238-7fac8c" podStartSLOduration=1.176958076 podStartE2EDuration="1.176958076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:14.170701692 +0000 UTC m=+1641.882227061" watchObservedRunningTime="2026-10-30 19:32:14.176958076 +0000 UTC m=+1641.888483443" Oct 30 19:32:14.356593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.356565 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:14.407428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.407392 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5b2016-4b60-4abb-bd42-633f299ddaee" path="/var/lib/kubelet/pods/3a5b2016-4b60-4abb-bd42-633f299ddaee/volumes" Oct 30 19:32:14.674055 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:14.674029 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8b87da_0a6e_4adf_aa47_4edbda0fd4f7.slice/crio-3150c474d799802b3bf667d9c08b387d910a4b5ad5b3eb3eb23ad0b3081939ef WatchSource:0}: Error finding container 3150c474d799802b3bf667d9c08b387d910a4b5ad5b3eb3eb23ad0b3081939ef: Status 404 returned error can't find the container with id 3150c474d799802b3bf667d9c08b387d910a4b5ad5b3eb3eb23ad0b3081939ef Oct 30 19:32:14.674896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:14.674867 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:32:15.086160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.086123 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9238-7fac8c"] Oct 30 19:32:15.092859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.092828 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9238-7fac8c"] Oct 30 19:32:15.150995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.150956 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" event={"ID":"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7","Type":"ContainerStarted","Data":"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe"} Oct 30 19:32:15.151218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.151038 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:15.151218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.151065 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" event={"ID":"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7","Type":"ContainerStarted","Data":"3150c474d799802b3bf667d9c08b387d910a4b5ad5b3eb3eb23ad0b3081939ef"} Oct 30 19:32:15.185544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.185496 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" podStartSLOduration=2.185482307 podStartE2EDuration="2.185482307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:15.184445444 +0000 UTC m=+1642.895970810" watchObservedRunningTime="2026-10-30 19:32:15.185482307 +0000 UTC m=+1642.897007703" Oct 30 19:32:15.266635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.264245 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9238-0c9587"] Oct 30 19:32:15.266635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.266047 4990 cpu_manager.go:414] "containerMap: removing container" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" containerName="test-curl-9238-7fac8c" Oct 30 19:32:15.266635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.266072 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" containerName="test-curl-9238-7fac8c" Oct 30 19:32:15.266635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.266400 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" containerName="test-curl-9238-7fac8c" Oct 30 19:32:15.272887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.272861 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:15.280337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.280307 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdqf\" (UniqueName: \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\") pod \"test-curl-9238-0c9587\" (UID: \"e4bd6be6-9d4e-40c8-9837-a083c0097319\") " pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:15.289333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.289308 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-0c9587"] Oct 30 19:32:15.382005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.381918 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdqf\" (UniqueName: \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\") pod \"test-curl-9238-0c9587\" (UID: \"e4bd6be6-9d4e-40c8-9837-a083c0097319\") " pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:15.399464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.399435 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdqf\" (UniqueName: \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\") pod \"test-curl-9238-0c9587\" (UID: \"e4bd6be6-9d4e-40c8-9837-a083c0097319\") " pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:15.590883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.590849 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:15.900924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:15.900896 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-0c9587"] Oct 30 19:32:15.902288 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:15.902260 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4bd6be6_9d4e_40c8_9837_a083c0097319.slice/crio-46a83f2ffa45d0dceb23c7d2f13c4e9ea5d3f25c66f5146488e951f5534057e3 WatchSource:0}: Error finding container 46a83f2ffa45d0dceb23c7d2f13c4e9ea5d3f25c66f5146488e951f5534057e3: Status 404 returned error can't find the container with id 46a83f2ffa45d0dceb23c7d2f13c4e9ea5d3f25c66f5146488e951f5534057e3 Oct 30 19:32:16.163141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:16.163012 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-0c9587" event={"ID":"e4bd6be6-9d4e-40c8-9837-a083c0097319","Type":"ContainerStarted","Data":"baf723d000f5c2577e2e05a35bcddb0902e100c234eb298878268358e2ac4cfe"} Oct 30 19:32:16.163141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:16.163055 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-0c9587" event={"ID":"e4bd6be6-9d4e-40c8-9837-a083c0097319","Type":"ContainerStarted","Data":"46a83f2ffa45d0dceb23c7d2f13c4e9ea5d3f25c66f5146488e951f5534057e3"} Oct 30 19:32:16.163393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:16.163280 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9238-7fac8c" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" containerName="test-curl-9238-7fac8c" containerID="cri-o://cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504" gracePeriod=2 Oct 30 19:32:16.198530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:16.198480 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9238-0c9587" podStartSLOduration=1.198466259 podStartE2EDuration="1.198466259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:16.194878364 +0000 UTC m=+1643.906403733" watchObservedRunningTime="2026-10-30 19:32:16.198466259 +0000 UTC m=+1643.909991625" Oct 30 19:32:16.206008 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:16.205974 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7fac8c\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:16.962450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:16.962395 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:17.095131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.085791 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9238-0c9587"] Oct 30 19:32:17.095131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.094124 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9238-0c9587"] Oct 30 19:32:17.128513 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.128480 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 19:32:17.259986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.259902 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9238-7e9019"] Oct 30 19:32:17.260682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.260663 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" containerName="test-curl-9238-0c9587" Oct 30 19:32:17.260682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.260681 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" containerName="test-curl-9238-0c9587" Oct 30 19:32:17.260825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.260799 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" containerName="test-curl-9238-0c9587" Oct 30 19:32:17.264491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.264476 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:17.299120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.297081 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-7e9019"] Oct 30 19:32:17.324466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.324427 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlfr\" (UniqueName: \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\") pod \"test-curl-9238-7e9019\" (UID: \"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372\") " pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:17.425798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.425753 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlfr\" (UniqueName: \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\") pod \"test-curl-9238-7e9019\" (UID: \"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372\") " pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:17.444305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.443736 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlfr\" (UniqueName: \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\") pod \"test-curl-9238-7e9019\" (UID: \"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372\") " pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:17.581076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.581045 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:17.757554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.757514 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:17.897605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:17.897569 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-7e9019"] Oct 30 19:32:17.899195 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:17.899165 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53fcfbc_d7ee_4170_8b6a_1a6c7b331372.slice/crio-162083c7ace5d2dfeeae90453931e2d487d6be59776090b2983ff020c68d2c2b WatchSource:0}: Error finding container 162083c7ace5d2dfeeae90453931e2d487d6be59776090b2983ff020c68d2c2b: Status 404 returned error can't find the container with id 162083c7ace5d2dfeeae90453931e2d487d6be59776090b2983ff020c68d2c2b Oct 30 19:32:18.191657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.191621 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-7e9019" event={"ID":"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372","Type":"ContainerStarted","Data":"e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9"} Oct 30 19:32:18.192034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.191677 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-7e9019" event={"ID":"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372","Type":"ContainerStarted","Data":"162083c7ace5d2dfeeae90453931e2d487d6be59776090b2983ff020c68d2c2b"} Oct 30 19:32:18.192034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.191857 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9238-0c9587" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" containerName="test-curl-9238-0c9587" containerID="cri-o://baf723d000f5c2577e2e05a35bcddb0902e100c234eb298878268358e2ac4cfe" gracePeriod=2 Oct 30 19:32:18.230606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.230556 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9238-7e9019" podStartSLOduration=1.230541062 podStartE2EDuration="1.230541062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:18.225828478 +0000 UTC m=+1645.937353845" watchObservedRunningTime="2026-10-30 19:32:18.230541062 +0000 UTC m=+1645.942066427" Oct 30 19:32:18.236981 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:18.236952 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-0c9587\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:18.372475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.372444 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:18.383510 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:18.383481 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-0c9587\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:18.466309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.466278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\" (UniqueName: \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\") pod \"11ed0526-2e95-4f11-a73d-03970c26a29f\" (UID: \"11ed0526-2e95-4f11-a73d-03970c26a29f\") " Oct 30 19:32:18.468644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.468615 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p" pod "11ed0526-2e95-4f11-a73d-03970c26a29f" (UID: "11ed0526-2e95-4f11-a73d-03970c26a29f"). InnerVolumeSpecName "kube-api-access-9xt6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:18.569258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:18.569227 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xt6p\" (UniqueName: \"kubernetes.io/projected/11ed0526-2e95-4f11-a73d-03970c26a29f-kube-api-access-9xt6p\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:32:19.100754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.100719 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9238-7e9019"] Oct 30 19:32:19.108926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.108895 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9238-7e9019"] Oct 30 19:32:19.204014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.203977 4990 generic.go:356] "Generic (PLEG): container finished" podID="11ed0526-2e95-4f11-a73d-03970c26a29f" containerID="cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504" exitCode=137 Oct 30 19:32:19.204486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.204028 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7fac8c" Oct 30 19:32:19.204486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.204072 4990 scope.go:122] "RemoveContainer" containerID="cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504" Oct 30 19:32:19.219065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.219049 4990 scope.go:122] "RemoveContainer" containerID="cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504" Oct 30 19:32:19.219363 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:19.219344 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504\": container with ID starting with cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504 not found: ID does not exist" containerID="cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504" Oct 30 19:32:19.219425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.219370 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504"} err="failed to get container status \"cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504\": rpc error: code = NotFound desc = could not find container \"cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504\": container with ID starting with cef36b1138285aa82ed10fef117c5926cc9376a1f4a0857fa384d14461245504 not found: ID does not exist" Oct 30 19:32:19.270775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.270746 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9238-5e0631"] Oct 30 19:32:19.271787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.271765 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" containerName="test-curl-9238-7e9019" Oct 30 19:32:19.271836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.271791 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" containerName="test-curl-9238-7e9019" Oct 30 19:32:19.272035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.272023 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" containerName="test-curl-9238-7e9019" Oct 30 19:32:19.277484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.277465 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:19.294987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.294511 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-5e0631"] Oct 30 19:32:19.382806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.382711 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsjq\" (UniqueName: \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\") pod \"test-curl-9238-5e0631\" (UID: \"d63cb223-c9c0-422e-b89b-e684de1370af\") " pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:19.485152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.485112 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsjq\" (UniqueName: \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\") pod \"test-curl-9238-5e0631\" (UID: \"d63cb223-c9c0-422e-b89b-e684de1370af\") " pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:19.503158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.503124 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsjq\" (UniqueName: \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\") pod \"test-curl-9238-5e0631\" (UID: \"d63cb223-c9c0-422e-b89b-e684de1370af\") " pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:19.531289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.531245 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.598117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.597231 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:19.928263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:19.928232 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9238-5e0631"] Oct 30 19:32:19.928869 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:19.928842 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd63cb223_c9c0_422e_b89b_e684de1370af.slice/crio-aaebd87eec13c3b8723f942868d4ffe5fa0014e08011544d16b06e93b0ec497a WatchSource:0}: Error finding container aaebd87eec13c3b8723f942868d4ffe5fa0014e08011544d16b06e93b0ec497a: Status 404 returned error can't find the container with id aaebd87eec13c3b8723f942868d4ffe5fa0014e08011544d16b06e93b0ec497a Oct 30 19:32:20.220705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.220663 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-5e0631" event={"ID":"d63cb223-c9c0-422e-b89b-e684de1370af","Type":"ContainerStarted","Data":"bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747"} Oct 30 19:32:20.221164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.220722 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9238-5e0631" event={"ID":"d63cb223-c9c0-422e-b89b-e684de1370af","Type":"ContainerStarted","Data":"aaebd87eec13c3b8723f942868d4ffe5fa0014e08011544d16b06e93b0ec497a"} Oct 30 19:32:20.227952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.227921 4990 generic.go:356] "Generic (PLEG): container finished" podID="e4bd6be6-9d4e-40c8-9837-a083c0097319" containerID="baf723d000f5c2577e2e05a35bcddb0902e100c234eb298878268358e2ac4cfe" exitCode=137 Oct 30 19:32:20.228166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.228119 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9238-7e9019" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" containerName="test-curl-9238-7e9019" containerID="cri-o://e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9" gracePeriod=2 Oct 30 19:32:20.253043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.252995 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9238-5e0631" podStartSLOduration=1.252982132 podStartE2EDuration="1.252982132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:20.252581631 +0000 UTC m=+1647.964106996" watchObservedRunningTime="2026-10-30 19:32:20.252982132 +0000 UTC m=+1647.964507497" Oct 30 19:32:20.263628 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:20.263595 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7e9019\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:20.359804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.359780 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:20.371267 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:20.371223 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7e9019\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:20.402889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.402856 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ed0526-2e95-4f11-a73d-03970c26a29f" path="/var/lib/kubelet/pods/11ed0526-2e95-4f11-a73d-03970c26a29f/volumes" Oct 30 19:32:20.501234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.501161 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\" (UniqueName: \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\") pod \"e4bd6be6-9d4e-40c8-9837-a083c0097319\" (UID: \"e4bd6be6-9d4e-40c8-9837-a083c0097319\") " Oct 30 19:32:20.503378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.503347 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf" pod "e4bd6be6-9d4e-40c8-9837-a083c0097319" (UID: "e4bd6be6-9d4e-40c8-9837-a083c0097319"). InnerVolumeSpecName "kube-api-access-vgdqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:20.603195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:20.603162 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdqf\" (UniqueName: \"kubernetes.io/projected/e4bd6be6-9d4e-40c8-9837-a083c0097319-kube-api-access-vgdqf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:32:21.030555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.030519 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7"] Oct 30 19:32:21.042337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.042309 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf20cd35-2dtq7"] Oct 30 19:32:21.106381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.106351 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9238-5e0631"] Oct 30 19:32:21.112408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.112378 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9238-5e0631"] Oct 30 19:32:21.214189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.206758 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:21.240218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.240192 4990 scope.go:122] "RemoveContainer" containerID="baf723d000f5c2577e2e05a35bcddb0902e100c234eb298878268358e2ac4cfe" Oct 30 19:32:21.240721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.240202 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-0c9587" Oct 30 19:32:21.902531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:21.902492 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.030769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.030736 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p"] Oct 30 19:32:22.042417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.042386 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d2bbc6-t6k9p"] Oct 30 19:32:22.062071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.062043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.250583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.250552 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9238-5e0631" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" containerName="test-curl-9238-5e0631" containerID="cri-o://bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747" gracePeriod=2 Oct 30 19:32:22.264743 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:22.264706 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-5e0631\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:22.371890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.371850 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:22.408703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.408668 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012339dd-35cc-4f80-99d9-c9091d69f6df" path="/var/lib/kubelet/pods/012339dd-35cc-4f80-99d9-c9091d69f6df/volumes" Oct 30 19:32:22.409526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.409501 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b0e10d-41af-454f-8e7c-fcc729e9fda7" path="/var/lib/kubelet/pods/08b0e10d-41af-454f-8e7c-fcc729e9fda7/volumes" Oct 30 19:32:22.410064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.410045 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bd6be6-9d4e-40c8-9837-a083c0097319" path="/var/lib/kubelet/pods/e4bd6be6-9d4e-40c8-9837-a083c0097319/volumes" Oct 30 19:32:22.412510 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:22.412479 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-5e0631\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:22.443829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.443806 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:22.456961 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:22.456930 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7e9019\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:22.538801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.538734 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\" (UniqueName: \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\") pod \"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372\" (UID: \"d53fcfbc-d7ee-4170-8b6a-1a6c7b331372\") " Oct 30 19:32:22.540864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.540835 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr" pod "d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" (UID: "d53fcfbc-d7ee-4170-8b6a-1a6c7b331372"). InnerVolumeSpecName "kube-api-access-qdlfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:22.640781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.640748 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdlfr\" (UniqueName: \"kubernetes.io/projected/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372-kube-api-access-qdlfr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:32:22.903722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:22.903688 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.157668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.157589 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.260829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.260793 4990 generic.go:356] "Generic (PLEG): container finished" podID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" containerID="e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9" exitCode=137 Oct 30 19:32:23.261243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.260851 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:23.261243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.260896 4990 scope.go:122] "RemoveContainer" containerID="e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9" Oct 30 19:32:23.272229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.272211 4990 scope.go:122] "RemoveContainer" containerID="e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9" Oct 30 19:32:23.272540 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:23.272520 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9\": container with ID starting with e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9 not found: ID does not exist" containerID="e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9" Oct 30 19:32:23.272644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.272549 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9"} err="failed to get container status \"e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9\": rpc error: code = NotFound desc = could not find container \"e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9\": container with ID starting with e2a2c1c88fb8f8854f5ad1d03d099f6d018e7cb88f1862dceb757166349199c9 not found: ID does not exist" Oct 30 19:32:23.274900 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:23.274878 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7e9019\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:23.288050 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:23.288024 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-7e9019\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" pod="opendatahub/test-curl-9238-7e9019" Oct 30 19:32:23.312569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:23.312532 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:24.407479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.407435 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53fcfbc-d7ee-4170-8b6a-1a6c7b331372" path="/var/lib/kubelet/pods/d53fcfbc-d7ee-4170-8b6a-1a6c7b331372/volumes" Oct 30 19:32:24.445262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.445240 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:24.576528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.576499 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\" (UniqueName: \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\") pod \"d63cb223-c9c0-422e-b89b-e684de1370af\" (UID: \"d63cb223-c9c0-422e-b89b-e684de1370af\") " Oct 30 19:32:24.578753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.578723 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq" pod "d63cb223-c9c0-422e-b89b-e684de1370af" (UID: "d63cb223-c9c0-422e-b89b-e684de1370af"). InnerVolumeSpecName "kube-api-access-bpsjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.620148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.620012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:24.679121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:24.679070 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpsjq\" (UniqueName: \"kubernetes.io/projected/d63cb223-c9c0-422e-b89b-e684de1370af-kube-api-access-bpsjq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:32:25.287069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:25.287031 4990 generic.go:356] "Generic (PLEG): container finished" podID="d63cb223-c9c0-422e-b89b-e684de1370af" containerID="bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747" exitCode=137 Oct 30 19:32:25.287334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:25.287145 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:25.287455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:25.287163 4990 scope.go:122] "RemoveContainer" containerID="bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747" Oct 30 19:32:25.304272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:25.304247 4990 scope.go:122] "RemoveContainer" containerID="bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747" Oct 30 19:32:25.304646 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:25.304625 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747\": container with ID starting with bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747 not found: ID does not exist" containerID="bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747" Oct 30 19:32:25.304715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:25.304657 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747"} err="failed to get container status \"bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747\": rpc error: code = NotFound desc = could not find container \"bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747\": container with ID starting with bc7a7281bb907924ef819298822d40fa888d1deee691db72606d0202168d3747 not found: ID does not exist" Oct 30 19:32:25.312293 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:25.312267 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-5e0631\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:26.179072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:26.179041 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:32:26.239861 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:26.237604 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9238-5e0631\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" pod="opendatahub/test-curl-9238-5e0631" Oct 30 19:32:26.405395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:26.405355 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" path="/var/lib/kubelet/pods/d63cb223-c9c0-422e-b89b-e684de1370af/volumes" Oct 30 19:32:27.044385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:27.044351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:28.789397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:28.789363 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.073866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:29.073823 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.392426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:29.392322 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:29.549630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:29.549598 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.050800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:31.050764 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:32.868993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:32.868960 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.691797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:34.691759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.845258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:34.845216 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:34.993819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:34.993721 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:35.097363 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:32:35.097325 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:32:36.924533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:36.924486 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:38.027102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.027056 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7"] Oct 30 19:32:38.036735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.036679 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aace3d5b-zklr7"] Oct 30 19:32:38.403903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.403868 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c90e98-389f-4305-8b36-dc46102d7170" path="/var/lib/kubelet/pods/c6c90e98-389f-4305-8b36-dc46102d7170/volumes" Oct 30 19:32:38.457959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.457920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:38.658328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.658209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:38.924765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:38.924675 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.070310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.070266 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.428155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.420912 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:32:39.428155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.425713 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" containerName="test-curl-9238-5e0631" Oct 30 19:32:39.428155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.425737 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" containerName="test-curl-9238-5e0631" Oct 30 19:32:39.428155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.425962 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63cb223-c9c0-422e-b89b-e684de1370af" containerName="test-curl-9238-5e0631" Oct 30 19:32:39.438197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.437165 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.451228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.451195 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:32:39.497119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.485568 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d9eb76-openshift-default-dockercfg-6vnqr\"" Oct 30 19:32:39.524813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.521580 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:39.551835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.551792 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.552260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.552229 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.552567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.552547 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.552762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.552743 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.555892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.555867 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.556061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.556043 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rz5\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.556276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.556259 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.556381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.556369 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.556572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.556559 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.657872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.657836 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.657930 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658001 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658081 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658129 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658153 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rz5\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658220 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658250 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658332 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658352 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658729 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.658901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.658878 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.659166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.659145 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.660782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.660756 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.661682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.661650 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.662266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.662238 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.680188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.680029 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rz5\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.681597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.681550 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:39.845497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:39.845464 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:40.213022 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:40.212985 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dec97f_23a4_47bd_980c_11fab281d90c.slice/crio-129309e821da3d3b624adefb1127ce307e39dfa03c595f4575a2133de71fef4b WatchSource:0}: Error finding container 129309e821da3d3b624adefb1127ce307e39dfa03c595f4575a2133de71fef4b: Status 404 returned error can't find the container with id 129309e821da3d3b624adefb1127ce307e39dfa03c595f4575a2133de71fef4b Oct 30 19:32:40.213886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.213839 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:32:40.216295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.216256 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:40.216396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.216329 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:40.216396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.216358 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:32:40.464492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.464390 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:40.539839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.539797 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerStarted","Data":"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a"} Oct 30 19:32:40.540002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.539863 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerStarted","Data":"129309e821da3d3b624adefb1127ce307e39dfa03c595f4575a2133de71fef4b"} Oct 30 19:32:40.590292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.590152 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podStartSLOduration=1.590131266 podStartE2EDuration="1.590131266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:40.585358958 +0000 UTC m=+1668.296884327" watchObservedRunningTime="2026-10-30 19:32:40.590131266 +0000 UTC m=+1668.301656632" Oct 30 19:32:40.708122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.708013 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:40.845540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.845512 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:40.845935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.845914 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:32:40.850439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.850421 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:40.858284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.858260 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:40.876814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.876780 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:32:40.887209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.887181 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-891d6b-dockercfg-x55ll\"" Oct 30 19:32:40.887443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.887426 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-891d6b\"" Oct 30 19:32:40.887963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.887935 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkct\" (UniqueName: \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:40.888216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:40.888194 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.041728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.006969 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.144165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.118368 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkct\" (UniqueName: \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.146419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.138723 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.226734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.224293 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkct\" (UniqueName: \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\") pod \"payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.241621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.241584 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:32:41.251235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.251141 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:32:41.251235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.251202 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.316996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.316960 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:41.350195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.350155 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxnq8\" (UniqueName: \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.350477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.350450 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.452483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.452446 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.452677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.452655 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxnq8\" (UniqueName: \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.453441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.453414 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.473632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.473595 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxnq8\" (UniqueName: \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\") pod \"payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.567913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.567880 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:41.644520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.642737 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:32:41.712306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.711909 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:41.851408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.851376 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:41.886753 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:41.886715 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b00a77_69da_4d93_a365_57dc0b8a0d74.slice/crio-0f90eac3b5062e327607f354bf0a170355544b5764572de8fc9b5ab3ef3c151d WatchSource:0}: Error finding container 0f90eac3b5062e327607f354bf0a170355544b5764572de8fc9b5ab3ef3c151d: Status 404 returned error can't find the container with id 0f90eac3b5062e327607f354bf0a170355544b5764572de8fc9b5ab3ef3c151d Oct 30 19:32:41.887924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:41.887897 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:32:42.559231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.559192 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" event={"ID":"030a0045-99a8-48e8-aba0-5a4af5b4838c","Type":"ContainerStarted","Data":"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c"} Oct 30 19:32:42.559726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.559251 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" event={"ID":"030a0045-99a8-48e8-aba0-5a4af5b4838c","Type":"ContainerStarted","Data":"7692b570c42336b9c02a3cf323218e3ab78fe35464689a2cce68270b94efd27b"} Oct 30 19:32:42.559726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.559273 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:42.561125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.561072 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" event={"ID":"10b00a77-69da-4d93-a365-57dc0b8a0d74","Type":"ContainerStarted","Data":"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7"} Oct 30 19:32:42.561263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.561151 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" event={"ID":"10b00a77-69da-4d93-a365-57dc0b8a0d74","Type":"ContainerStarted","Data":"0f90eac3b5062e327607f354bf0a170355544b5764572de8fc9b5ab3ef3c151d"} Oct 30 19:32:42.561263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.561240 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:42.561346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.561279 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:42.562207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.562187 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:32:42.600687 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.600633 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" podStartSLOduration=2.6006170170000003 podStartE2EDuration="2.600617017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:42.596855643 +0000 UTC m=+1670.308381010" watchObservedRunningTime="2026-10-30 19:32:42.600617017 +0000 UTC m=+1670.312142384" Oct 30 19:32:42.636532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:42.636484 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" podStartSLOduration=2.636472264 podStartE2EDuration="2.636472264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:42.634299521 +0000 UTC m=+1670.345824898" watchObservedRunningTime="2026-10-30 19:32:42.636472264 +0000 UTC m=+1670.347997630" Oct 30 19:32:43.307429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:43.307389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.453016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:43.452962 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.607319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:43.607232 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.949939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:43.946452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:45.651337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:45.651290 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:45.915935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:45.915849 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:46.191180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:46.168298 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:46.196600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:46.196569 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65"] Oct 30 19:32:46.212501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:46.212454 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f07f9926-kzx65"] Oct 30 19:32:46.474159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:46.459030 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3401f092-7c60-4eaf-b44d-e84f803a41bf" path="/var/lib/kubelet/pods/3401f092-7c60-4eaf-b44d-e84f803a41bf/volumes" Oct 30 19:32:47.650303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:47.648342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:48.873557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:48.873516 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.339672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:49.339631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.698129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:49.698033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:50.450335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:50.450293 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:52.090770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:52.090737 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:52.231057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:52.231015 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:53.195477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:53.195438 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:53.350904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:53.350861 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:53.510164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:53.510033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:53.573615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:53.573589 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:32:53.573788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:53.573663 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:32:54.161620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.161578 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:54.751273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.751241 4990 scope.go:122] "RemoveContainer" containerID="71a30857ed8e0ebe179ed218a84245ee27ce6fe2982f3bf0ad5de5a8f8ed9173" Oct 30 19:32:54.762717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.762689 4990 scope.go:122] "RemoveContainer" containerID="7172f8867aa6b41c4222c3032d0c56560d5abddf8d2f64e2da8f418bda3b1955" Oct 30 19:32:54.772465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.772443 4990 scope.go:122] "RemoveContainer" containerID="fb62533cbd513b3311d65f20d1294da3df37a908626519af1451fc69d202d63e" Oct 30 19:32:54.781839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.781818 4990 scope.go:122] "RemoveContainer" containerID="cb9d808a1ad844d2b7f641c6e7211fec23eb8e858e1fc1aa4f4b5a3e6f74dc4d" Oct 30 19:32:54.791294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.791272 4990 scope.go:122] "RemoveContainer" containerID="5ce6947aafdf1c0a776db9f17b7db7378007ef878b9bea32426a1c981385bfe9" Oct 30 19:32:54.801137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.801119 4990 scope.go:122] "RemoveContainer" containerID="6f9477fa35ff5dccbc4a2c4dfc3bd265f06600be83d3f7fbbbd4646c88cea607" Oct 30 19:32:54.974230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:54.974189 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:55.107906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.107872 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 19:32:55.114872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.114850 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.131611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.131582 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:55.136344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.136319 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 19:32:55.169554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.169512 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cd4bde\"" Oct 30 19:32:55.171495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.171461 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cd4bde-dockercfg-cp7vc\"" Oct 30 19:32:55.394169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.391476 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.394169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.391684 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfvg\" (UniqueName: \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.494077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.494041 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.494359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.494238 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfvg\" (UniqueName: \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.494998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.494945 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.519067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.519039 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfvg\" (UniqueName: \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\") pod \"payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:55.730680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:55.730598 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:56.039695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.039664 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 19:32:56.040853 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:32:56.040822 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6fb294_f2d4_4061_b0bc_3ff24ef42a2b.slice/crio-13024b5c40bdf14c2846b6638de85adc20202569dcbd999d335d6055d964a53e WatchSource:0}: Error finding container 13024b5c40bdf14c2846b6638de85adc20202569dcbd999d335d6055d964a53e: Status 404 returned error can't find the container with id 13024b5c40bdf14c2846b6638de85adc20202569dcbd999d335d6055d964a53e Oct 30 19:32:56.237125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.236378 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:56.626298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.626237 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:56.780129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.780077 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:56.926657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.926566 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" event={"ID":"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b","Type":"ContainerStarted","Data":"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9"} Oct 30 19:32:56.926657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.926629 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" event={"ID":"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b","Type":"ContainerStarted","Data":"13024b5c40bdf14c2846b6638de85adc20202569dcbd999d335d6055d964a53e"} Oct 30 19:32:56.926657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.926658 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:32:56.964775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:56.964722 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" podStartSLOduration=1.96470841 podStartE2EDuration="1.96470841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:56.960997756 +0000 UTC m=+1684.672523122" watchObservedRunningTime="2026-10-30 19:32:56.96470841 +0000 UTC m=+1684.676233788" Oct 30 19:32:57.032364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:57.032331 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65"] Oct 30 19:32:57.037103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:57.037070 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6622202d-wxs65"] Oct 30 19:32:58.100340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:58.100280 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:58.402825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:58.402742 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361387d9-0cdf-43ac-880a-03a602c863df" path="/var/lib/kubelet/pods/361387d9-0cdf-43ac-880a-03a602c863df/volumes" Oct 30 19:32:59.244492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:59.244449 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.803305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:59.803271 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.982991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:32:59.982952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:01.711908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:01.711860 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:02.029391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:02.029308 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj"] Oct 30 19:33:02.039063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:02.039034 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa97429-c92xj"] Oct 30 19:33:02.405001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:02.404964 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813d44f2-c20c-42b4-8bf5-0c71f8eb94f4" path="/var/lib/kubelet/pods/813d44f2-c20c-42b4-8bf5-0c71f8eb94f4/volumes" Oct 30 19:33:03.386991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:03.386948 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:05.089509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:05.089468 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:05.115678 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:33:05.115641 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:33:06.031012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:06.030976 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq"] Oct 30 19:33:06.040443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:06.040419 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b78259-p7lgq"] Oct 30 19:33:06.406493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:06.406451 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0be76c-4f11-4c95-9393-a414f92f1ac5" path="/var/lib/kubelet/pods/8c0be76c-4f11-4c95-9393-a414f92f1ac5/volumes" Oct 30 19:33:06.897353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:06.897316 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.050154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:07.050086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.187074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:07.186984 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.832306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:07.832269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:07.939331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:07.939298 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 19:33:09.318658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:09.318611 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:11.057188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:11.057086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:12.250871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:12.250829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:13.831115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:13.825292 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:14.113415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:14.086692 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:14.827733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:14.827675 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 19:33:14.849043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:14.839209 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:14.888274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:14.888230 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:14.966280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:14.966230 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.120610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.120523 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw"] Oct 30 19:33:15.120914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.120882 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.154116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.152204 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f149a23-ltrqw"] Oct 30 19:33:15.292382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.292341 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 19:33:15.439625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.439497 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.554690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.554646 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.695180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.689504 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Oct 30 19:33:15.712442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.712395 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.778210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.778167 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nlrv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:15.778499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.778473 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.829116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.828184 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" podUID="052bfe48-6728-4136-93e2-7090d3d90481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.829116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.828738 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:33:15.853115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.847284 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.853115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.852550 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:33:15.974212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.930404 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 19:33:15.974212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.932014 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:33:15.974212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:15.932351 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:33:16.036134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.014106 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Oct 30 19:33:16.036134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.024604 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.119622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.119487 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:16.312879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.312809 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.314166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.314115 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:33:16.504239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.504191 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.590370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.586813 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.761013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.760973 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.797182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.797145 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f663a90-512e-484b-a5cd-d4ec8626dfdf" path="/var/lib/kubelet/pods/2f663a90-512e-484b-a5cd-d4ec8626dfdf/volumes" Oct 30 19:33:16.798111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.798019 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:33:16.798444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.798429 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 19:33:16.798734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.798719 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp" Oct 30 19:33:16.798871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.798859 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:33:16.799008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.798997 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 19:33:16.799143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.799132 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 19:33:16.799270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.799257 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:33:16.799447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.799396 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:33:16.809353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.809074 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 19:33:16.824683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:16.823788 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 19:33:17.806770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:17.806494 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:18.486115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:18.483975 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:18.696781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:18.696742 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:18.881315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:18.876465 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:19.384685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:19.384649 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:20.029930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:20.029893 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6"] Oct 30 19:33:20.038212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:20.038182 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62efc1c8-wgzv6"] Oct 30 19:33:20.409234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:20.409191 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7562cbd-7cf6-4bd5-8468-2bac1599726e" path="/var/lib/kubelet/pods/c7562cbd-7cf6-4bd5-8468-2bac1599726e/volumes" Oct 30 19:33:20.630043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:20.630006 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:21.818367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:21.818329 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:22.532875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:22.532834 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:23.962964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.962922 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerID="c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7" exitCode=0 Oct 30 19:33:23.963443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.962985 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerDied","Data":"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7"} Oct 30 19:33:23.963490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.963450 4990 scope.go:122] "RemoveContainer" containerID="c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7" Oct 30 19:33:23.965324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.965289 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:23.965423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.965382 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:23.965482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:23.965428 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:24.129779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:24.126429 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:25.003511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.003418 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerStarted","Data":"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea"} Oct 30 19:33:25.096197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.093692 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:33:25.108087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.108055 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:33:25.109741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.109708 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:25.109897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.109772 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:25.799306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.797631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:25.961642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:25.961609 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:26.039908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.039871 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj"] Oct 30 19:33:26.049376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.049301 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5a955-5s9gj"] Oct 30 19:33:26.107870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.107815 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:26.108049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.107886 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:26.212430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.212394 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:33:26.213663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.213631 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" containerID="cri-o://f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961" gracePeriod=30 Oct 30 19:33:26.264364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.264328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:26.461916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.461836 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef04b2e-ff69-45d7-87f3-b5d637e4a45e" path="/var/lib/kubelet/pods/fef04b2e-ff69-45d7-87f3-b5d637e4a45e/volumes" Oct 30 19:33:26.723204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.723081 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:26.903010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:26.902973 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:27.056414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:27.056382 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:27.110935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:27.110897 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:27.111123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:27.110954 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:27.814728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:27.814690 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:28.108198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:28.108107 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:28.108198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:28.108175 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:29.119127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:29.117713 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:29.119127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:29.117779 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:29.647714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:29.647677 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:30.108643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:30.108566 4990 patch_prober.go:29] interesting pod/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:33:30.108643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:30.108626 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:33:31.530963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:31.530890 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:33:31.536164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:31.534775 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:33:31.901614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:31.901483 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:32.014136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.010633 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:33:32.147218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147118 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147229 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147272 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147320 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147372 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147489 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147551 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147683 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.147840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.147821 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\") pod \"1972b098-4e10-475a-bffd-256073373f8c\" (UID: \"1972b098-4e10-475a-bffd-256073373f8c\") " Oct 30 19:33:32.148221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148189 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:32.148475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148451 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:32.148620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148599 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.148705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148624 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1972b098-4e10-475a-bffd-256073373f8c-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.148705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148632 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:32.148836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148809 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:32.148909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.148828 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:32.151243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.151215 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:32.151516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.151491 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:32.151611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.151577 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:32.152147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.152074 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk" pod "1972b098-4e10-475a-bffd-256073373f8c" (UID: "1972b098-4e10-475a-bffd-256073373f8c"). InnerVolumeSpecName "kube-api-access-x84xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:32.169132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.167500 4990 generic.go:356] "Generic (PLEG): container finished" podID="1972b098-4e10-475a-bffd-256073373f8c" containerID="f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961" exitCode=0 Oct 30 19:33:32.169132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.168723 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" Oct 30 19:33:32.175122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.174479 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerDied","Data":"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961"} Oct 30 19:33:32.175122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.174574 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4" event={"ID":"1972b098-4e10-475a-bffd-256073373f8c","Type":"ContainerDied","Data":"9d705598605169c77e1c102a175ff70499f80e265f5cb7d60956713f781e9e6b"} Oct 30 19:33:32.175122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.174605 4990 scope.go:122] "RemoveContainer" containerID="f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961" Oct 30 19:33:32.214598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.214576 4990 scope.go:122] "RemoveContainer" containerID="cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9" Oct 30 19:33:32.225103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.225071 4990 scope.go:122] "RemoveContainer" containerID="f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961" Oct 30 19:33:32.225385 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:33:32.225362 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961\": container with ID starting with f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961 not found: ID does not exist" containerID="f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961" Oct 30 19:33:32.225630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.225392 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961"} err="failed to get container status \"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961\": rpc error: code = NotFound desc = could not find container \"f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961\": container with ID starting with f96288115d6ea5025c3bb7f07faff4dc22dab833806fd010cf21cca213f30961 not found: ID does not exist" Oct 30 19:33:32.225630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.225409 4990 scope.go:122] "RemoveContainer" containerID="cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9" Oct 30 19:33:32.225743 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:33:32.225653 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9\": container with ID starting with cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9 not found: ID does not exist" containerID="cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9" Oct 30 19:33:32.225743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.225668 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9"} err="failed to get container status \"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9\": rpc error: code = NotFound desc = could not find container \"cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9\": container with ID starting with cd179598db46c7e83c177575d2ff59001f34d566aa83c8ba5a0d3f35936094b9 not found: ID does not exist" Oct 30 19:33:32.242498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.242469 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:33:32.247744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.247722 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-87b0dcce-openshift-default-7fc555fd6f-55wh4"] Oct 30 19:33:32.250328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250296 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250324 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250335 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1972b098-4e10-475a-bffd-256073373f8c-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250345 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84xk\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-kube-api-access-x84xk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250354 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1972b098-4e10-475a-bffd-256073373f8c-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250363 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.250464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.250370 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1972b098-4e10-475a-bffd-256073373f8c-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:33:32.409796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:32.409711 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1972b098-4e10-475a-bffd-256073373f8c" path="/var/lib/kubelet/pods/1972b098-4e10-475a-bffd-256073373f8c/volumes" Oct 30 19:33:33.519028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:33.518995 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:33.731251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:33.731188 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:33.954454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:33.954424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:34.095029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.094993 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:34.424632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.424600 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:33:34.425946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.425917 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:34.426158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.426133 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:34.426402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.426385 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:34.431004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.430986 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.447788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.447761 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:33:34.451122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.451077 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-092bc0-openshift-default-dockercfg-z28jc\"" Oct 30 19:33:34.516806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.516530 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:34.518081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518043 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518121 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518151 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518210 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxnw\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518248 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518393 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518543 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518630 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.518705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.518677 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.628556 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.628717 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.628859 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.628932 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.628978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.629084 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.629145 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.629169 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.629893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.629226 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxnw\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.630057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.630020 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.666115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.630302 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.704469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.679548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.704469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.680595 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.704469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.681113 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.716122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.715695 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.717005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.716826 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxnw\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.717005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.716910 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.747830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.747752 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\") pod \"e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.755702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.755665 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:34.999126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:34.993658 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:35.233153 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:33:35.232869 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:33:35.263005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:35.262972 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:33:35.264784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:33:35.264754 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa79589c_0eec_43ad_ab32_03644bd98b51.slice/crio-e3bd77e4e5ac3bf32d81bfc11264180c614dcbde331a2937979b0928df03fc39 WatchSource:0}: Error finding container e3bd77e4e5ac3bf32d81bfc11264180c614dcbde331a2937979b0928df03fc39: Status 404 returned error can't find the container with id e3bd77e4e5ac3bf32d81bfc11264180c614dcbde331a2937979b0928df03fc39 Oct 30 19:33:35.290157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:35.278694 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:35.290157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:35.278792 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:35.290157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:35.278835 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:35.842950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:35.842596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:36.234027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:36.233922 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerStarted","Data":"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805"} Oct 30 19:33:36.234027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:36.233994 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerStarted","Data":"e3bd77e4e5ac3bf32d81bfc11264180c614dcbde331a2937979b0928df03fc39"} Oct 30 19:33:36.287115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:36.286772 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podStartSLOduration=2.2867543169999998 podStartE2EDuration="2.286754317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:36.283575126 +0000 UTC m=+1723.995100530" watchObservedRunningTime="2026-10-30 19:33:36.286754317 +0000 UTC m=+1723.998279684" Oct 30 19:33:36.758107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:36.756604 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:36.763137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:36.762528 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:37.029938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:37.029852 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9"] Oct 30 19:33:37.034294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:37.034269 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2db0d4-gn2l9"] Oct 30 19:33:37.242946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:37.242910 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:37.244132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:37.244110 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:33:37.832226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:37.832132 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:38.405876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:38.405835 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66436e1a-eca6-42a9-bfd3-4d1a570b212b" path="/var/lib/kubelet/pods/66436e1a-eca6-42a9-bfd3-4d1a570b212b/volumes" Oct 30 19:33:38.696001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:38.695915 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:38.850723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:38.850685 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:39.731858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.731815 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 19:33:39.744150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.733519 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:39.744403 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.744376 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:39.744853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.744827 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1972b098-4e10-475a-bffd-256073373f8c" containerName="istio-proxy" Oct 30 19:33:39.769464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.769433 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 19:33:39.770800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.770671 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791527 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791592 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791621 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791778 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzf6\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.791947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791890 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.792005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.791951 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.792060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.792008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.792155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.792081 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.893978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894073 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894231 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894266 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzf6\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894314 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894360 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894405 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894464 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894527 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.894667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894579 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.895081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894822 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.895081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.894991 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.895081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.895048 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.897410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.897384 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.897864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.897835 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.912701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.912678 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzf6\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:39.917934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:39.917906 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:40.101332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.101297 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:40.156320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.154108 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:40.521438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.519938 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 19:33:40.524824 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:33:40.524785 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6b9ecf_8d12_4f2c_abfe_3cd079fe1a1d.slice/crio-2afcf85d1bf217a83abaec8becaa2b8743dde3a0f35316cbea604c061f4c79cd WatchSource:0}: Error finding container 2afcf85d1bf217a83abaec8becaa2b8743dde3a0f35316cbea604c061f4c79cd: Status 404 returned error can't find the container with id 2afcf85d1bf217a83abaec8becaa2b8743dde3a0f35316cbea604c061f4c79cd Oct 30 19:33:40.528692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.528655 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:40.528788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.528737 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:40.528866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:40.528786 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:41.307455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:41.307411 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" event={"ID":"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d","Type":"ContainerStarted","Data":"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047"} Oct 30 19:33:41.308008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:41.307511 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" event={"ID":"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d","Type":"ContainerStarted","Data":"2afcf85d1bf217a83abaec8becaa2b8743dde3a0f35316cbea604c061f4c79cd"} Oct 30 19:33:41.934568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:41.934526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.054125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.054062 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podStartSLOduration=3.054047524 podStartE2EDuration="3.054047524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:41.370358367 +0000 UTC m=+1729.081883735" watchObservedRunningTime="2026-10-30 19:33:42.054047524 +0000 UTC m=+1729.765572890" Oct 30 19:33:42.054890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.054866 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:33:42.058530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.058513 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.077134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.076385 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-092bc0\"" Oct 30 19:33:42.100113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.098852 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-092bc0-dockercfg-2nd4d\"" Oct 30 19:33:42.100992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.100959 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.101892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.101429 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:42.104118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.103217 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:33:42.104118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.103266 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:33:42.105184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.105157 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:33:42.133671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.133607 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94b4\" (UniqueName: \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.133885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.133750 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.288167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.271305 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94b4\" (UniqueName: \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.288167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.271483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.288167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.272552 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.292187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.291669 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:33:42.435107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.412460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.447117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.440166 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94b4\" (UniqueName: \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\") pod \"payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.447117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.444918 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:33:42.448760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.447470 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.454813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.454402 4990 generic.go:356] "Generic (PLEG): container finished" podID="72dec97f-23a4-47bd-980c-11fab281d90c" containerID="496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a" exitCode=0 Oct 30 19:33:42.464196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.464163 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerDied","Data":"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a"} Oct 30 19:33:42.464698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.464672 4990 scope.go:122] "RemoveContainer" containerID="496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a" Oct 30 19:33:42.539930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.539757 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:42.539930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.539877 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:42.540269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.539933 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:33:42.541755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.541585 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwrn\" (UniqueName: \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.541755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.541694 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.562034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.562012 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:33:42.645002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.644967 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwrn\" (UniqueName: \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.645686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.645662 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.646870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.646700 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.738277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.738246 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:42.739397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.739353 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwrn\" (UniqueName: \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\") pod \"payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:42.843488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:42.843451 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:43.125339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.121526 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:33:43.125339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.121593 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:33:43.141826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.141081 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:33:43.349206 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:33:43.349056 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73bf729_2ded_4316_ac13_1222f4990fa7.slice/crio-4ffeb5e3083c75e518fd8550b143124f4c41b6cdbc80c11538de79a01ffa786e WatchSource:0}: Error finding container 4ffeb5e3083c75e518fd8550b143124f4c41b6cdbc80c11538de79a01ffa786e: Status 404 returned error can't find the container with id 4ffeb5e3083c75e518fd8550b143124f4c41b6cdbc80c11538de79a01ffa786e Oct 30 19:33:43.351534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.351406 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:33:43.506252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.497880 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" event={"ID":"c73bf729-2ded-4316-ac13-1222f4990fa7","Type":"ContainerStarted","Data":"4ffeb5e3083c75e518fd8550b143124f4c41b6cdbc80c11538de79a01ffa786e"} Oct 30 19:33:43.508530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.508482 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerStarted","Data":"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d"} Oct 30 19:33:43.518982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.514229 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" event={"ID":"9c0bea28-7338-4e78-a206-c2c791367b02","Type":"ContainerStarted","Data":"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683"} Oct 30 19:33:43.519177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.519159 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:43.519326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.519308 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" event={"ID":"9c0bea28-7338-4e78-a206-c2c791367b02","Type":"ContainerStarted","Data":"966728ef2b30184ac4c3c1c7b320489ca5a06c4e08c995f7bd710075e6c62ce1"} Oct 30 19:33:43.596351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.596287 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" podStartSLOduration=1.596266429 podStartE2EDuration="1.596266429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:43.589520107 +0000 UTC m=+1731.301045473" watchObservedRunningTime="2026-10-30 19:33:43.596266429 +0000 UTC m=+1731.307791795" Oct 30 19:33:43.846509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.846478 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:33:43.847723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.847697 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:43.847824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:43.847743 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:44.102719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.102630 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:33:44.102990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.102962 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:33:44.208199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.208164 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:44.540058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.539944 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" event={"ID":"c73bf729-2ded-4316-ac13-1222f4990fa7","Type":"ContainerStarted","Data":"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c"} Oct 30 19:33:44.595114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.591280 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" podStartSLOduration=2.591262616 podStartE2EDuration="2.591262616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:44.588382374 +0000 UTC m=+1732.299907741" watchObservedRunningTime="2026-10-30 19:33:44.591262616 +0000 UTC m=+1732.302787986" Oct 30 19:33:44.857278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.856837 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:44.857278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:44.856918 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:45.031153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.031071 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w"] Oct 30 19:33:45.035802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.035767 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0636b68-wd52w"] Oct 30 19:33:45.108174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.107599 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:33:45.108174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.107660 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:33:45.549402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.549374 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:45.856172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.855211 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:45.856172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:45.855272 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:46.115286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.101924 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.159117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.153746 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:33:46.430646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.430139 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bce047-2cb6-479a-91e6-c0fde409e640" path="/var/lib/kubelet/pods/e4bce047-2cb6-479a-91e6-c0fde409e640/volumes" Oct 30 19:33:46.674112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.664776 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:46.896116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.894136 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.896116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:46.894219 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:47.317865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:47.317828 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:47.319685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:47.319664 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:47.325558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:47.325539 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 19:33:47.845741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:47.845707 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:47.846193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:47.845759 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:48.845986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:48.845954 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:48.846413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:48.846011 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:48.890215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:48.890176 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.534720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:49.534685 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.846648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:49.846616 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:33:49.847054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:49.846665 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:33:50.039534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.039500 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s"] Oct 30 19:33:50.048334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.048288 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09429ac-8j67s"] Oct 30 19:33:50.403788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.403753 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f939a409-1671-4646-9219-63d9d9b6f337" path="/var/lib/kubelet/pods/f939a409-1671-4646-9219-63d9d9b6f337/volumes" Oct 30 19:33:50.855037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.855005 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:33:50.856659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.856631 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:33:50.857350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.857324 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:33:50.930304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:50.930268 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:52.862139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:52.861174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.187309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.187270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.458258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.458169 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.547230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.547200 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:33:54.851914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.851869 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.913539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.913496 4990 scope.go:122] "RemoveContainer" containerID="9101a423d241e1f76db9da7e60d7fbe278d5e7682ddfee929a727f69de65d5b8" Oct 30 19:33:54.928284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.928260 4990 scope.go:122] "RemoveContainer" containerID="6d561427d7e396bcd5c331a1dedd4f21aa58923235d211b13b4d4e1c41b80a79" Oct 30 19:33:54.941582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.941463 4990 scope.go:122] "RemoveContainer" containerID="e6383711dfcaefdabd96cfa0a816486d84b066e4ef03ee124c2b6b5fa2a19d2e" Oct 30 19:33:54.960146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.960120 4990 scope.go:122] "RemoveContainer" containerID="c8fcf9f6053105c6a4256ee8d24eda2a475463b984899c59855b705cb6dc9127" Oct 30 19:33:54.977498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.977468 4990 scope.go:122] "RemoveContainer" containerID="47d0267d7e39981cfcf52f912684b6d9643a97250dcad00e7d05d87d1ccba4ff" Oct 30 19:33:54.996581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:54.996557 4990 scope.go:122] "RemoveContainer" containerID="282d56c2b6921777ac268a9a1463843a67b52abe7002765c1daebc9c62703dfb" Oct 30 19:33:55.006951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:55.006929 4990 scope.go:122] "RemoveContainer" containerID="20d906e327d24a8edbae678d1cbe1f7a316ec90b1d29dca48fd4676273e82ed2" Oct 30 19:33:55.016338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:55.016318 4990 scope.go:122] "RemoveContainer" containerID="2f1b3d15b9d053ad5dbb82407024d6477293f6507c9fce8985377e73872c6955" Oct 30 19:33:56.563348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:56.563312 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:33:56.692660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:56.692612 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:58.445954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:58.444352 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.320291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:59.320206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.470337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:59.470296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.776351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:33:59.776250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:00.239379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:00.239342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:01.088849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:01.088809 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:01.546408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:01.546306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:02.014899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:02.014857 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:02.155026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:02.154987 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:03.883834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:03.883743 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.038261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:05.038211 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.098559 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:05.098514 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:34:05.523792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:05.523757 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.686138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:05.685968 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:05.938436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:05.938389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:06.098691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:06.098629 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:06.247384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:06.247296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:06.404607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:06.404567 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.057943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:08.057906 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.225817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:08.225785 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.486384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:08.475610 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.634908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:08.634870 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:08.961808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:08.961767 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:09.845067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.845033 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:34:09.850699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.850675 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.905853 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.905930 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.905980 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906059 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906115 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906149 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906182 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906261 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj28\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.926388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.906403 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:09.937366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.936949 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:34:09.938222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:09.937992 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.008578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008543 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008657 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008699 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008818 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008857 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008889 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.008957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.008919 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009009 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj28\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009190 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009207 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009531 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.009721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.009697 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.011421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.011401 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.011731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.011711 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.031631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.031607 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj28\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.032801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.032780 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.210051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.209975 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:10.441392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.440148 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.843452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.843381 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:34:10.844890 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:10.844858 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2dac39_9736_4673_b47f_52ddbd76c017.slice/crio-e35a94217a98801e5d3331e2728213941e2e9f18d59cfea4964b45eec85eaf07 WatchSource:0}: Error finding container e35a94217a98801e5d3331e2728213941e2e9f18d59cfea4964b45eec85eaf07: Status 404 returned error can't find the container with id e35a94217a98801e5d3331e2728213941e2e9f18d59cfea4964b45eec85eaf07 Oct 30 19:34:10.847944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.847907 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:10.848286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.847975 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:10.848286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.848006 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:10.962753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.962713 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" event={"ID":"9d2dac39-9736-4673-b47f-52ddbd76c017","Type":"ContainerStarted","Data":"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382"} Oct 30 19:34:10.962938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:10.962782 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" event={"ID":"9d2dac39-9736-4673-b47f-52ddbd76c017","Type":"ContainerStarted","Data":"e35a94217a98801e5d3331e2728213941e2e9f18d59cfea4964b45eec85eaf07"} Oct 30 19:34:11.010654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.010180 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podStartSLOduration=2.010160973 podStartE2EDuration="2.010160973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:10.999960754 +0000 UTC m=+1758.711486119" watchObservedRunningTime="2026-10-30 19:34:11.010160973 +0000 UTC m=+1758.721686341" Oct 30 19:34:11.021877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.021841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.205017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.204929 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:11.210483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.210454 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:11.212623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.212596 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:11.212724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:11.212650 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:12.210858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:12.210824 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:12.211316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:12.210888 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:13.199112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:13.198829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:13.215347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:13.215298 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:13.216065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:13.216030 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:14.210661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:14.210628 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:14.210827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:14.210691 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:15.082730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:15.082689 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:15.210621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:15.210589 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:15.210824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:15.210658 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:16.210774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:16.210734 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:34:16.211212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:16.210796 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:34:17.254448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:17.254406 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:17.256650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:17.255515 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:17.258891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:17.258864 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:34:17.545744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:17.545633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:17.910267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:17.910230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:18.195704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:18.195606 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:19.627072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.627012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:19.787758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.787710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:19.878196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.878118 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:34:19.881860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.881838 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:19.904658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.904625 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:34:19.906348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.906324 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7d9e82-dockercfg-bh28k\"" Oct 30 19:34:19.942006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.941975 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:19.963372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.963338 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7d9e82\"" Oct 30 19:34:19.988169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.980395 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:19.988169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.980611 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdksn\" (UniqueName: \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:19.988169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:19.984318 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:34:20.032616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.032581 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.096494 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.096639 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdksn\" (UniqueName: \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:20.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.096896 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5cl\" (UniqueName: \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.096981 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:20.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.098002 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:20.218196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.216214 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.218196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.216481 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5cl\" (UniqueName: \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.239860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.236106 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.239860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.239750 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5cl\" (UniqueName: \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\") pod \"payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.273225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.273164 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:34:20.285197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.285131 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdksn\" (UniqueName: \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\") pod \"payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:20.404611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.404575 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:20.497707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.497632 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:20.724242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.724185 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:34:20.725741 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:20.725633 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0790539f_a33a_4715_b04a_cc82ac4caa11.slice/crio-9761c06ac86c8479ad338a2ba7664d0a90238d20662ef8b581d46e2e882e4c0b WatchSource:0}: Error finding container 9761c06ac86c8479ad338a2ba7664d0a90238d20662ef8b581d46e2e882e4c0b: Status 404 returned error can't find the container with id 9761c06ac86c8479ad338a2ba7664d0a90238d20662ef8b581d46e2e882e4c0b Oct 30 19:34:20.835147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:20.835113 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:34:20.836578 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:20.836545 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a84b5e6_1a10_4cae_86f7_e3cf589c147f.slice/crio-2cf2a19c71de437fac7e5d01c9c8d1642e8961ec914b97c45a018076bc59aeb7 WatchSource:0}: Error finding container 2cf2a19c71de437fac7e5d01c9c8d1642e8961ec914b97c45a018076bc59aeb7: Status 404 returned error can't find the container with id 2cf2a19c71de437fac7e5d01c9c8d1642e8961ec914b97c45a018076bc59aeb7 Oct 30 19:34:21.244490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.244450 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" event={"ID":"6a84b5e6-1a10-4cae-86f7-e3cf589c147f","Type":"ContainerStarted","Data":"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248"} Oct 30 19:34:21.244696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.244520 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:21.244696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.244549 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" event={"ID":"6a84b5e6-1a10-4cae-86f7-e3cf589c147f","Type":"ContainerStarted","Data":"2cf2a19c71de437fac7e5d01c9c8d1642e8961ec914b97c45a018076bc59aeb7"} Oct 30 19:34:21.246414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.246387 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" event={"ID":"0790539f-a33a-4715-b04a-cc82ac4caa11","Type":"ContainerStarted","Data":"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72"} Oct 30 19:34:21.246549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.246434 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" event={"ID":"0790539f-a33a-4715-b04a-cc82ac4caa11","Type":"ContainerStarted","Data":"9761c06ac86c8479ad338a2ba7664d0a90238d20662ef8b581d46e2e882e4c0b"} Oct 30 19:34:21.246549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.246495 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:21.281955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.281907 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" podStartSLOduration=2.28189001 podStartE2EDuration="2.28189001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:21.277817588 +0000 UTC m=+1768.989342936" watchObservedRunningTime="2026-10-30 19:34:21.28189001 +0000 UTC m=+1768.993415376" Oct 30 19:34:21.315234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.315188 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" podStartSLOduration=2.315174164 podStartE2EDuration="2.315174164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:21.311357169 +0000 UTC m=+1769.022882534" watchObservedRunningTime="2026-10-30 19:34:21.315174164 +0000 UTC m=+1769.026699529" Oct 30 19:34:21.405065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:21.405032 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:23.210685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:23.210646 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:24.677983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:24.677922 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:24.910323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:24.910221 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:25.114624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:25.114568 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.428305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.428263 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.501692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.501658 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 19:34:26.505377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.505358 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.526496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.526470 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 19:34:26.530824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.530804 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-287ea2\"" Oct 30 19:34:26.602103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.602055 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 19:34:26.609121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.609079 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:26.672138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.650539 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-287ea2-dockercfg-7mh9p\"" Oct 30 19:34:26.672138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.665641 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.672138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.665965 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkb5r\" (UniqueName: \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.896526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.894321 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.901714 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.903915 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkb5r\" (UniqueName: \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.903965 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpsf\" (UniqueName: \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.904036 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.904146 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.916948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.905152 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.942174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.935068 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkb5r\" (UniqueName: \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\") pod \"payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:26.987759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.987726 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:34:26.987972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:26.987927 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" containerID="cri-o://ecf78cbe34512b37d08dbf8cf4cad53f62122ef2b3b9fadd72a4967721b1efcb" gracePeriod=30 Oct 30 19:34:27.007000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.006304 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpsf\" (UniqueName: \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:27.007000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.006394 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:27.009009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.008976 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:27.027034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.026998 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpsf\" (UniqueName: \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\") pod \"payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:27.120353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.120315 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:27.259878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.259842 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:27.438957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.438921 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:27.448921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.448893 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 19:34:27.451784 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:27.451749 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f79183a_38ca_4be3_ae0d_b5ca9cb1cba0.slice/crio-c0b005b4fcb78035d3f8ada687b82edde4564824ba5d9031b6fb8767ea327ebf WatchSource:0}: Error finding container c0b005b4fcb78035d3f8ada687b82edde4564824ba5d9031b6fb8767ea327ebf: Status 404 returned error can't find the container with id c0b005b4fcb78035d3f8ada687b82edde4564824ba5d9031b6fb8767ea327ebf Oct 30 19:34:27.595630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:27.595604 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 19:34:27.596927 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:27.596901 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a9a932_101a_4cdd_bd86_759cb4892678.slice/crio-7918afdec811076b84f6d645212624ec8c33a3754acea702997f5e5aab0d043d WatchSource:0}: Error finding container 7918afdec811076b84f6d645212624ec8c33a3754acea702997f5e5aab0d043d: Status 404 returned error can't find the container with id 7918afdec811076b84f6d645212624ec8c33a3754acea702997f5e5aab0d043d Oct 30 19:34:28.320617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.320579 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" event={"ID":"e6a9a932-101a-4cdd-bd86-759cb4892678","Type":"ContainerStarted","Data":"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca"} Oct 30 19:34:28.320844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.320644 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" event={"ID":"e6a9a932-101a-4cdd-bd86-759cb4892678","Type":"ContainerStarted","Data":"7918afdec811076b84f6d645212624ec8c33a3754acea702997f5e5aab0d043d"} Oct 30 19:34:28.320844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.320707 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:28.322609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.322572 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" event={"ID":"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0","Type":"ContainerStarted","Data":"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb"} Oct 30 19:34:28.322755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.322622 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" event={"ID":"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0","Type":"ContainerStarted","Data":"c0b005b4fcb78035d3f8ada687b82edde4564824ba5d9031b6fb8767ea327ebf"} Oct 30 19:34:28.322755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.322696 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:28.361896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.361835 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" podStartSLOduration=2.361817036 podStartE2EDuration="2.361817036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:28.355404917 +0000 UTC m=+1776.066930287" watchObservedRunningTime="2026-10-30 19:34:28.361817036 +0000 UTC m=+1776.073342403" Oct 30 19:34:28.389845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:28.389797 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" podStartSLOduration=2.389784572 podStartE2EDuration="2.389784572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:28.389454808 +0000 UTC m=+1776.100980174" watchObservedRunningTime="2026-10-30 19:34:28.389784572 +0000 UTC m=+1776.101309958" Oct 30 19:34:29.554351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:29.554313 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:31.277816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:31.277780 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.117640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.117192 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:34:32.117640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.117438 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" containerID="cri-o://599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c" gracePeriod=30 Oct 30 19:34:32.133498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.133466 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:34:32.133722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.133699 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" containerID="cri-o://72c1b2d5ef435977893d1caea66928bb77e6f928dc9e9efa016020ac8a6fa01e" gracePeriod=30 Oct 30 19:34:32.140684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.140333 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:34:32.140684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.140512 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" containerID="cri-o://169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863" gracePeriod=30 Oct 30 19:34:32.145693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.145665 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:34:32.145869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.145850 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" containerID="cri-o://aeac255e7e05459a6f1eb16b0c1f7b6dad083d2ff7399a8691afbe2d4549d68a" gracePeriod=30 Oct 30 19:34:32.310691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.309243 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:34:32.311768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.311744 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:34:32.513524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.513485 4990 generic.go:356] "Generic (PLEG): container finished" podID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerID="ecf78cbe34512b37d08dbf8cf4cad53f62122ef2b3b9fadd72a4967721b1efcb" exitCode=0 Oct 30 19:34:32.514210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.514188 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" event={"ID":"3ce1795b-beee-4956-99f2-dc7fbbb89954","Type":"ContainerDied","Data":"ecf78cbe34512b37d08dbf8cf4cad53f62122ef2b3b9fadd72a4967721b1efcb"} Oct 30 19:34:32.514422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.514401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" event={"ID":"3ce1795b-beee-4956-99f2-dc7fbbb89954","Type":"ContainerDied","Data":"25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf"} Oct 30 19:34:32.514551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.514535 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25fb318c0c75a707b20e01d57efa0587671e5383e737e1e8af6bc54c4a851cbf" Oct 30 19:34:32.519235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.519207 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.593637 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.593771 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.593836 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.593976 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.594046 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.594194 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.594248 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.594371 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.607166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.594448 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\") pod \"3ce1795b-beee-4956-99f2-dc7fbbb89954\" (UID: \"3ce1795b-beee-4956-99f2-dc7fbbb89954\") " Oct 30 19:34:32.617523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.617484 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.619250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.619174 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.619464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.619445 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:32.620885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.620859 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.621246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.621212 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.633128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.633104 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "kube-api-access-8rjq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.633446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.633428 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.635303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.635277 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.639067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.637651 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo" pod "3ce1795b-beee-4956-99f2-dc7fbbb89954" (UID: "3ce1795b-beee-4956-99f2-dc7fbbb89954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:32.704871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.704832 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705116 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705238 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705343 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705426 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705515 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705654 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce1795b-beee-4956-99f2-dc7fbbb89954-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705762 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce1795b-beee-4956-99f2-dc7fbbb89954-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.705875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:32.705860 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjq4\" (UniqueName: \"kubernetes.io/projected/3ce1795b-beee-4956-99f2-dc7fbbb89954-kube-api-access-8rjq4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.043511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:33.043477 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:33.524002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:33.523969 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d" Oct 30 19:34:33.602740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:33.602700 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:34:33.608902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:33.608872 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-8sl7d"] Oct 30 19:34:34.406346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:34.406311 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" path="/var/lib/kubelet/pods/3ce1795b-beee-4956-99f2-dc7fbbb89954/volumes" Oct 30 19:34:34.711206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:34.710648 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:35.100542 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:35.100502 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:34:36.403112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:36.402411 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:37.523116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.521555 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:34:37.556620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.556575 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:34:37.580122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.577367 4990 generic.go:356] "Generic (PLEG): container finished" podID="c2351615-9d67-4c7c-a594-f4038175a585" containerID="599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c" exitCode=0 Oct 30 19:34:37.580122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.577497 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerDied","Data":"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c"} Oct 30 19:34:37.580122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.577547 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" event={"ID":"c2351615-9d67-4c7c-a594-f4038175a585","Type":"ContainerDied","Data":"86c01d91debd850496731bbdd58a97da74feeca632f3d56578fa00bbb43c68d7"} Oct 30 19:34:37.580122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.577575 4990 scope.go:122] "RemoveContainer" containerID="599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c" Oct 30 19:34:37.580122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.577738 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8" Oct 30 19:34:37.607534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.605723 4990 generic.go:356] "Generic (PLEG): container finished" podID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerID="169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863" exitCode=0 Oct 30 19:34:37.607534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.605852 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" event={"ID":"08c5e9ad-2f0a-42fe-8812-289bcec10cf7","Type":"ContainerDied","Data":"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863"} Oct 30 19:34:37.607534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.605913 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" event={"ID":"08c5e9ad-2f0a-42fe-8812-289bcec10cf7","Type":"ContainerDied","Data":"6c5c8686a7c6d29958e11f3363fb4f2aa65c14932b87bfd5fe00c74886be7cf6"} Oct 30 19:34:37.607534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.606004 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6" Oct 30 19:34:37.607534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.607458 4990 scope.go:122] "RemoveContainer" containerID="08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72" Oct 30 19:34:37.610412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.609989 4990 generic.go:356] "Generic (PLEG): container finished" podID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerID="72c1b2d5ef435977893d1caea66928bb77e6f928dc9e9efa016020ac8a6fa01e" exitCode=0 Oct 30 19:34:37.610412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.610213 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" event={"ID":"713fa0d3-f937-42e1-a3a7-fab259226bc2","Type":"ContainerDied","Data":"72c1b2d5ef435977893d1caea66928bb77e6f928dc9e9efa016020ac8a6fa01e"} Oct 30 19:34:37.616693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.616568 4990 generic.go:356] "Generic (PLEG): container finished" podID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerID="aeac255e7e05459a6f1eb16b0c1f7b6dad083d2ff7399a8691afbe2d4549d68a" exitCode=0 Oct 30 19:34:37.616693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.616610 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" event={"ID":"e7652dfa-4468-4eeb-ac34-b160116cae61","Type":"ContainerDied","Data":"aeac255e7e05459a6f1eb16b0c1f7b6dad083d2ff7399a8691afbe2d4549d68a"} Oct 30 19:34:37.625408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.625386 4990 scope.go:122] "RemoveContainer" containerID="599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c" Oct 30 19:34:37.625737 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:37.625708 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c\": container with ID starting with 599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c not found: ID does not exist" containerID="599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c" Oct 30 19:34:37.625833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.625747 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c"} err="failed to get container status \"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c\": rpc error: code = NotFound desc = could not find container \"599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c\": container with ID starting with 599be7a3a553d25e8f6d0e78199523b6b70e05d8da9f16fc9b9cbe900506283c not found: ID does not exist" Oct 30 19:34:37.625833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.625771 4990 scope.go:122] "RemoveContainer" containerID="08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72" Oct 30 19:34:37.626679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626658 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.626776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626703 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.626776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626753 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.626882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626823 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.626941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626888 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.626999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.626964 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627058 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627109 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.627210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627176 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.627265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627230 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627274 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627325 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.627370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627356 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.627478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627413 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627472 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627509 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.627685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627669 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\") pod \"c2351615-9d67-4c7c-a594-f4038175a585\" (UID: \"c2351615-9d67-4c7c-a594-f4038175a585\") " Oct 30 19:34:37.627751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.627738 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\") pod \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\" (UID: \"08c5e9ad-2f0a-42fe-8812-289bcec10cf7\") " Oct 30 19:34:37.628935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.628651 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:34:37.629395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.629211 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:37.629395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.629307 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "kube-api-access-66d8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.629744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.629722 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.629998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.629981 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.630176 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.630388 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.630601 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:37.630730 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72\": container with ID starting with 08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72 not found: ID does not exist" containerID="08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.630764 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72"} err="failed to get container status \"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72\": rpc error: code = NotFound desc = could not find container \"08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72\": container with ID starting with 08302a95870e4ff7ecf4ab916857bbd9577fce8a20c1e77ec51e6909be20cc72 not found: ID does not exist" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.630787 4990 scope.go:122] "RemoveContainer" containerID="169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.631012 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.631380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.631323 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:37.632805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.632400 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.632805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.632589 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.632805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.632695 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.633330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.633293 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.633875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.633843 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.634656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.634621 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.634831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.634700 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:37.634831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.634700 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo" pod "08c5e9ad-2f0a-42fe-8812-289bcec10cf7" (UID: "08c5e9ad-2f0a-42fe-8812-289bcec10cf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:37.634944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.634878 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4" pod "c2351615-9d67-4c7c-a594-f4038175a585" (UID: "c2351615-9d67-4c7c-a594-f4038175a585"). InnerVolumeSpecName "kube-api-access-6tbz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.639212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.637341 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:34:37.646075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.646055 4990 scope.go:122] "RemoveContainer" containerID="169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863" Oct 30 19:34:37.646474 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:37.646445 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863\": container with ID starting with 169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863 not found: ID does not exist" containerID="169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863" Oct 30 19:34:37.646591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.646483 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863"} err="failed to get container status \"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863\": rpc error: code = NotFound desc = could not find container \"169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863\": container with ID starting with 169fde383c64d704e43806efb9fbd92cd25ee19ede649ff2ef3044bb909a4863 not found: ID does not exist" Oct 30 19:34:37.729273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729186 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.729273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729243 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.729522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729330 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.729628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729600 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729739 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729784 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729839 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729870 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.729993 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730076 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730141 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730186 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730253 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730287 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730346 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730373 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730408 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730441 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730572 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\") pod \"e7652dfa-4468-4eeb-ac34-b160116cae61\" (UID: \"e7652dfa-4468-4eeb-ac34-b160116cae61\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730658 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.730711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.730699 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\") pod \"713fa0d3-f937-42e1-a3a7-fab259226bc2\" (UID: \"713fa0d3-f937-42e1-a3a7-fab259226bc2\") " Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731646 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731670 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-kube-api-access-66d8k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731689 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731706 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731719 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731733 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731748 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2351615-9d67-4c7c-a594-f4038175a585-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.731762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731761 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731776 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731789 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731801 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731817 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731830 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2351615-9d67-4c7c-a594-f4038175a585-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731842 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731864 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731876 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731890 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731902 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2351615-9d67-4c7c-a594-f4038175a585-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731915 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731928 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tbz4\" (UniqueName: \"kubernetes.io/projected/c2351615-9d67-4c7c-a594-f4038175a585-kube-api-access-6tbz4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.731940 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c5e9ad-2f0a-42fe-8812-289bcec10cf7-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.732279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.732257 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:37.733685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.732510 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.733685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.732698 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.733685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.733038 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.733685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.733257 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.733685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.733564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.733964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.733899 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:37.735872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.735574 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.736643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.736615 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "kube-api-access-cq4hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.736902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.736869 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.737416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.737389 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.737943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.737903 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "kube-api-access-g84mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:37.738484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.738452 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:37.738781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.738753 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo" pod "713fa0d3-f937-42e1-a3a7-fab259226bc2" (UID: "713fa0d3-f937-42e1-a3a7-fab259226bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:37.739914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.739891 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo" pod "e7652dfa-4468-4eeb-ac34-b160116cae61" (UID: "e7652dfa-4468-4eeb-ac34-b160116cae61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:37.833231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833197 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833225 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833234 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833243 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4hn\" (UniqueName: \"kubernetes.io/projected/713fa0d3-f937-42e1-a3a7-fab259226bc2-kube-api-access-cq4hn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833252 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833260 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833269 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652dfa-4468-4eeb-ac34-b160116cae61-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833276 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833285 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833292 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833303 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84mq\" (UniqueName: \"kubernetes.io/projected/e7652dfa-4468-4eeb-ac34-b160116cae61-kube-api-access-g84mq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833311 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833321 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713fa0d3-f937-42e1-a3a7-fab259226bc2-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833329 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713fa0d3-f937-42e1-a3a7-fab259226bc2-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.833526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.833336 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652dfa-4468-4eeb-ac34-b160116cae61-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:37.930343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.930316 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:34:37.935933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.935905 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-vqvp8"] Oct 30 19:34:37.964145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.964118 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:34:37.971425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:37.971393 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9l9v6"] Oct 30 19:34:38.288173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.287231 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:34:38.288173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.287326 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:34:38.404521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.404440 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" path="/var/lib/kubelet/pods/08c5e9ad-2f0a-42fe-8812-289bcec10cf7/volumes" Oct 30 19:34:38.405067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.405052 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2351615-9d67-4c7c-a594-f4038175a585" path="/var/lib/kubelet/pods/c2351615-9d67-4c7c-a594-f4038175a585/volumes" Oct 30 19:34:38.605037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.605003 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:38.629992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.629958 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" Oct 30 19:34:38.630187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.630011 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758" event={"ID":"713fa0d3-f937-42e1-a3a7-fab259226bc2","Type":"ContainerDied","Data":"79f8c24508e2b98a37a2b52f0b43781a8c3a80def5d7144d7061d0210ea12ec2"} Oct 30 19:34:38.630187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.630066 4990 scope.go:122] "RemoveContainer" containerID="72c1b2d5ef435977893d1caea66928bb77e6f928dc9e9efa016020ac8a6fa01e" Oct 30 19:34:38.631987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.631823 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" event={"ID":"e7652dfa-4468-4eeb-ac34-b160116cae61","Type":"ContainerDied","Data":"e5e35f43044fed4edcc2049a9af439817cd06724e2e5d83f88077937f30fd0dc"} Oct 30 19:34:38.631987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.631908 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm" Oct 30 19:34:38.642219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.642201 4990 scope.go:122] "RemoveContainer" containerID="aeac255e7e05459a6f1eb16b0c1f7b6dad083d2ff7399a8691afbe2d4549d68a" Oct 30 19:34:38.677983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.677914 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:34:38.682279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.682254 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-6g758"] Oct 30 19:34:38.714587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.714555 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:34:38.719368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:38.719342 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-pf2fm"] Oct 30 19:34:39.341131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.341061 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 19:34:39.341349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.341170 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 19:34:39.645270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.645185 4990 generic.go:356] "Generic (PLEG): container finished" podID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerID="86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805" exitCode=0 Oct 30 19:34:39.645678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.645257 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerDied","Data":"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805"} Oct 30 19:34:39.645760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.645738 4990 scope.go:122] "RemoveContainer" containerID="86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805" Oct 30 19:34:39.647828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.647799 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:39.647917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.647879 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:39.647917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:39.647911 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:34:40.022280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.022168 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.407510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.407473 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" path="/var/lib/kubelet/pods/713fa0d3-f937-42e1-a3a7-fab259226bc2/volumes" Oct 30 19:34:40.409288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.408589 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" path="/var/lib/kubelet/pods/e7652dfa-4468-4eeb-ac34-b160116cae61/volumes" Oct 30 19:34:40.657714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.657632 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerStarted","Data":"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7"} Oct 30 19:34:40.756120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.756059 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:34:40.756291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.756213 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:40.756291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:40.756259 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:41.757057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:41.757025 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:41.757464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:41.757080 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:41.917049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:41.917011 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.756340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:42.756305 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:42.756565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:42.756375 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:43.325112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:43.325051 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.756929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:43.756841 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:43.756929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:43.756902 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:43.789120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:43.784244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.756447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:44.756408 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:44.756841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:44.756471 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:45.045018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:45.044853 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.513288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:45.513200 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.723777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:45.723735 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.756852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:45.756818 4990 patch_prober.go:29] interesting pod/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:34:45.757383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:45.756884 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 19:34:46.211354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.211300 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9244-29d7a8"] Oct 30 19:34:46.212559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212527 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" Oct 30 19:34:46.212559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212556 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212616 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212625 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212647 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212656 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212677 4990 cpu_manager.go:414] "containerMap: removing container" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212685 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" Oct 30 19:34:46.212733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212729 4990 cpu_manager.go:414] "containerMap: removing container" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" Oct 30 19:34:46.212948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212737 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" Oct 30 19:34:46.212948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212910 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:46.212948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212928 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c5e9ad-2f0a-42fe-8812-289bcec10cf7" containerName="istio-proxy" Oct 30 19:34:46.212948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212941 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:46.213213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212952 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="713fa0d3-f937-42e1-a3a7-fab259226bc2" containerName="istio-proxy" Oct 30 19:34:46.213213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212964 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7652dfa-4468-4eeb-ac34-b160116cae61" containerName="istio-proxy" Oct 30 19:34:46.213213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.212985 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce1795b-beee-4956-99f2-dc7fbbb89954" containerName="istio-proxy" Oct 30 19:34:46.217041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.217022 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:46.243684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.243190 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:34:46.247740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.247709 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9244-29d7a8"] Oct 30 19:34:46.269252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.267511 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.355154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.333246 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jjx\" (UniqueName: \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\") pod \"test-curl-9244-29d7a8\" (UID: \"e76f5d59-819e-4c6a-a86f-59d4e358dda9\") " pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:46.465595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.465404 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.472211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.472180 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jjx\" (UniqueName: \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\") pod \"test-curl-9244-29d7a8\" (UID: \"e76f5d59-819e-4c6a-a86f-59d4e358dda9\") " pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:46.499741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.499706 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jjx\" (UniqueName: \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\") pod \"test-curl-9244-29d7a8\" (UID: \"e76f5d59-819e-4c6a-a86f-59d4e358dda9\") " pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:46.551852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.551800 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:46.769400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.763696 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:34:46.769400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.767376 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:34:46.770126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.770041 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:34:46.923005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:46.922223 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9244-29d7a8"] Oct 30 19:34:46.925679 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:46.925650 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode76f5d59_819e_4c6a_a86f_59d4e358dda9.slice/crio-fc72ce66379c332fafa0bafa1285152c40e8abebab945b1a008cd0076db6d77e WatchSource:0}: Error finding container fc72ce66379c332fafa0bafa1285152c40e8abebab945b1a008cd0076db6d77e: Status 404 returned error can't find the container with id fc72ce66379c332fafa0bafa1285152c40e8abebab945b1a008cd0076db6d77e Oct 30 19:34:47.641463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:47.641426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:47.784493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:47.784457 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9244-29d7a8" event={"ID":"e76f5d59-819e-4c6a-a86f-59d4e358dda9","Type":"ContainerStarted","Data":"7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e"} Oct 30 19:34:47.788322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:47.788291 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9244-29d7a8" event={"ID":"e76f5d59-819e-4c6a-a86f-59d4e358dda9","Type":"ContainerStarted","Data":"fc72ce66379c332fafa0bafa1285152c40e8abebab945b1a008cd0076db6d77e"} Oct 30 19:34:47.807082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:47.807048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:47.823375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:47.823327 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9244-29d7a8" podStartSLOduration=1.823312455 podStartE2EDuration="1.823312455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:47.816934829 +0000 UTC m=+1795.528460269" watchObservedRunningTime="2026-10-30 19:34:47.823312455 +0000 UTC m=+1795.534837821" Oct 30 19:34:48.194960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:48.194923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.358126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:48.357539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.844407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:48.844369 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9244-29d7a8"] Oct 30 19:34:48.850822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:48.850790 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9244-29d7a8"] Oct 30 19:34:49.877610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:49.877554 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.035673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:50.035625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.200921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:50.200825 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.662152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:50.662112 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.840277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:50.833714 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.846766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:50.846720 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9244-29d7a8" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" containerName="test-curl-9244-29d7a8" containerID="cri-o://7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e" gracePeriod=2 Oct 30 19:34:52.161772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:52.161736 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:52.823557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:52.823521 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:52.974617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:52.974416 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:53.066598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.066571 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:53.210506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.210383 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\" (UniqueName: \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\") pod \"e76f5d59-819e-4c6a-a86f-59d4e358dda9\" (UID: \"e76f5d59-819e-4c6a-a86f-59d4e358dda9\") " Oct 30 19:34:53.212643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.212615 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx" pod "e76f5d59-819e-4c6a-a86f-59d4e358dda9" (UID: "e76f5d59-819e-4c6a-a86f-59d4e358dda9"). InnerVolumeSpecName "kube-api-access-t4jjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:53.221702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.221676 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:53.312569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.312529 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jjx\" (UniqueName: \"kubernetes.io/projected/e76f5d59-819e-4c6a-a86f-59d4e358dda9-kube-api-access-t4jjx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:34:53.384276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.384247 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:34:53.384982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.384966 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" containerName="test-curl-9244-29d7a8" Oct 30 19:34:53.385041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.384983 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" containerName="test-curl-9244-29d7a8" Oct 30 19:34:53.385041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.385012 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:53.385041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.385017 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2351615-9d67-4c7c-a594-f4038175a585" containerName="istio-proxy" Oct 30 19:34:53.385164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.385153 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" containerName="test-curl-9244-29d7a8" Oct 30 19:34:53.388891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.388874 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.414802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.414778 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b625eb-dockercfg-dq6nz\"" Oct 30 19:34:53.414987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.414975 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b625eb\"" Oct 30 19:34:53.440654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.440613 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:34:53.575171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.547730 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttbs\" (UniqueName: \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.575483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.575449 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.575985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.575952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:53.664114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.636785 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:34:53.690163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.668286 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.690163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.684959 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttbs\" (UniqueName: \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.690163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.685030 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.690163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.686326 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.700452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.700423 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:34:53.727541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.727467 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttbs\" (UniqueName: \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\") pod \"payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:53.786695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.786657 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6zm\" (UniqueName: \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.786870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.786715 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.878223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.878084 4990 generic.go:356] "Generic (PLEG): container finished" podID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" containerID="7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e" exitCode=137 Oct 30 19:34:53.878223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.878162 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:53.878223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.878143 4990 scope.go:122] "RemoveContainer" containerID="7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e" Oct 30 19:34:53.888056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.888033 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6zm\" (UniqueName: \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.888250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.888104 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.888992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.888969 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:53.891074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.891056 4990 scope.go:122] "RemoveContainer" containerID="7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e" Oct 30 19:34:53.891379 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:53.891352 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e\": container with ID starting with 7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e not found: ID does not exist" containerID="7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e" Oct 30 19:34:53.891441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.891389 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e"} err="failed to get container status \"7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e\": rpc error: code = NotFound desc = could not find container \"7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e\": container with ID starting with 7e60a7c6dbfec8ade235017e86588ca543293e088748e8372b743a756a39c98e not found: ID does not exist" Oct 30 19:34:53.902059 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:34:53.902029 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9244-29d7a8\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" pod="opendatahub/test-curl-9244-29d7a8" Oct 30 19:34:53.906044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:53.906015 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6zm\" (UniqueName: \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\") pod \"payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:54.003887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.003841 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:54.059197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.059162 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:54.122570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.122539 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:34:54.129731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.129670 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:34:54.141146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.140927 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:34:54.154587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.154558 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:34:54.328828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.328795 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:34:54.329949 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:54.329924 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2ac32_e3ea_434f_b883_39cb87d74a86.slice/crio-040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f WatchSource:0}: Error finding container 040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f: Status 404 returned error can't find the container with id 040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f Oct 30 19:34:54.351697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.351664 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:54.427208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.427171 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76f5d59-819e-4c6a-a86f-59d4e358dda9" path="/var/lib/kubelet/pods/e76f5d59-819e-4c6a-a86f-59d4e358dda9/volumes" Oct 30 19:34:54.427691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.427661 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:34:54.889314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.889273 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" event={"ID":"37a2ac32-e3ea-434f-b883-39cb87d74a86","Type":"ContainerStarted","Data":"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b"} Oct 30 19:34:54.889520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.889329 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" event={"ID":"37a2ac32-e3ea-434f-b883-39cb87d74a86","Type":"ContainerStarted","Data":"040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f"} Oct 30 19:34:54.889520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.889358 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:34:54.892389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.892353 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" event={"ID":"7f666317-1012-4e64-9078-0301067d1603","Type":"ContainerStarted","Data":"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9"} Oct 30 19:34:54.892504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.892409 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" event={"ID":"7f666317-1012-4e64-9078-0301067d1603","Type":"ContainerStarted","Data":"821a464c3a67ce2cb6b702dfcb50dfdde00c09a9d58013f9b79ce78167a64ab5"} Oct 30 19:34:54.892504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.892440 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:34:54.929799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.929753 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" podStartSLOduration=1.929741816 podStartE2EDuration="1.929741816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:54.929077183 +0000 UTC m=+1802.640602548" watchObservedRunningTime="2026-10-30 19:34:54.929741816 +0000 UTC m=+1802.641267181" Oct 30 19:34:54.969873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:54.969824 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" podStartSLOduration=1.969810284 podStartE2EDuration="1.969810284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:54.967350696 +0000 UTC m=+1802.678876062" watchObservedRunningTime="2026-10-30 19:34:54.969810284 +0000 UTC m=+1802.681335692" Oct 30 19:34:55.102999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.102965 4990 scope.go:122] "RemoveContainer" containerID="df623b9b373f015bb0376f7c3b3b27ea0a8e47445037cd83d4c93a15bf34a2dd" Oct 30 19:34:55.119197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.119174 4990 scope.go:122] "RemoveContainer" containerID="dfb3f3a3c8153c00a6e6b413923836a669facab627b666abb47249b8cdf4acb0" Oct 30 19:34:55.130975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.130950 4990 scope.go:122] "RemoveContainer" containerID="774133175bf819fd02025426b28a2cdccb9736cb4d2ab741b0e9c016960b9d59" Oct 30 19:34:55.145670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.145638 4990 scope.go:122] "RemoveContainer" containerID="8cc3c8554e41743925ca7b11016c449b8eaea3f259976f3f35c12db76ee55a3d" Oct 30 19:34:55.160811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.160699 4990 scope.go:122] "RemoveContainer" containerID="ecf78cbe34512b37d08dbf8cf4cad53f62122ef2b3b9fadd72a4967721b1efcb" Oct 30 19:34:55.189420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.189393 4990 scope.go:122] "RemoveContainer" containerID="e20827e44232284ed19496acaf7b144efd8dfe5c25ce4801c630e18c6361822c" Oct 30 19:34:55.212017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.211657 4990 scope.go:122] "RemoveContainer" containerID="f132208c8b949ead689f8c0a21df3cae145d74b537abb3d84484d9dcfe8fd30c" Oct 30 19:34:55.228646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.228624 4990 scope.go:122] "RemoveContainer" containerID="42d36cbf2c5bf7322ca09f1ac557f4a13580cc2611eb7706a0cc0a463aa2695a" Oct 30 19:34:55.239380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.239358 4990 scope.go:122] "RemoveContainer" containerID="b017fc42dc56734776864bf00f84364a6401579814e8d401e978500daf9b1011" Oct 30 19:34:55.287353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.287078 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.439302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.439109 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.519829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.519783 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:34:55.535799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.535750 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.538891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.538861 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:34:55.552141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.551324 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9c8a2a\"" Oct 30 19:34:55.552141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.551436 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9c8a2a-dockercfg-zm7tt\"" Oct 30 19:34:55.615918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.615882 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.616114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.616048 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27kg\" (UniqueName: \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.717523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.717427 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.717714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.717564 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27kg\" (UniqueName: \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.718273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.718251 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.737171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.737085 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27kg\" (UniqueName: \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\") pod \"payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:55.866915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:55.866886 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:56.192261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.192223 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:34:56.194719 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:34:56.194689 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2e06e8_cf86_409f_b493_e261a7eb1f28.slice/crio-3c2b6b72fa93ab771d19055f19c4f03b681994ff752c3c609d7d8e5632d01069 WatchSource:0}: Error finding container 3c2b6b72fa93ab771d19055f19c4f03b681994ff752c3c609d7d8e5632d01069: Status 404 returned error can't find the container with id 3c2b6b72fa93ab771d19055f19c4f03b681994ff752c3c609d7d8e5632d01069 Oct 30 19:34:56.529072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.528972 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:56.921950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.921915 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" event={"ID":"0a2e06e8-cf86-409f-b493-e261a7eb1f28","Type":"ContainerStarted","Data":"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3"} Oct 30 19:34:56.922177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.921996 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" event={"ID":"0a2e06e8-cf86-409f-b493-e261a7eb1f28","Type":"ContainerStarted","Data":"3c2b6b72fa93ab771d19055f19c4f03b681994ff752c3c609d7d8e5632d01069"} Oct 30 19:34:56.922177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.922061 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:34:56.960792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:56.960741 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" podStartSLOduration=1.960729307 podStartE2EDuration="1.960729307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:56.95925961 +0000 UTC m=+1804.670784989" watchObservedRunningTime="2026-10-30 19:34:56.960729307 +0000 UTC m=+1804.672254672" Oct 30 19:34:58.029629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:58.029592 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks"] Oct 30 19:34:58.041325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:58.041288 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba6f29aa-mxzks"] Oct 30 19:34:58.408415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:58.408375 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df37ff8a-875d-4405-9fe6-b2bfb757d6cb" path="/var/lib/kubelet/pods/df37ff8a-875d-4405-9fe6-b2bfb757d6cb/volumes" Oct 30 19:34:58.447750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:58.447710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.489416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:34:59.489384 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.323645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:00.323605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.488174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:00.488065 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.803007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:00.802972 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:35:01.001530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:01.001496 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:01.151889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:01.151779 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:02.975508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:02.975468 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.737672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:04.737572 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.103625 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:05.103581 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:35:05.804262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:05.804224 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" containerID="cri-o://a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe" gracePeriod=25 Oct 30 19:35:05.909985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:05.909957 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:35:05.910190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:05.910041 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:35:06.165385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:06.165300 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:35:06.431728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:06.431633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:07.934288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:07.934255 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:35:08.351645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:08.351605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.797483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:10.797378 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:13.158177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:13.158138 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.299942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:15.299901 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.164868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:16.164823 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:35:17.435014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.434978 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:35:17.435531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.435270 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" containerID="cri-o://7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73" gracePeriod=30 Oct 30 19:35:17.447360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.447332 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:35:17.447637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.447586 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" containerID="cri-o://7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31" gracePeriod=30 Oct 30 19:35:17.570853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.570808 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:17.614009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.612688 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:35:17.614009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.612945 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" containerID="cri-o://1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab" gracePeriod=30 Oct 30 19:35:17.617657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.617618 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:35:17.618161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:17.618059 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" containerID="cri-o://da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340" gracePeriod=30 Oct 30 19:35:19.398882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:19.398828 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:21.339746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:21.339711 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.691314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.690605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.819961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.819931 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:35:22.829863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.829830 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:35:22.909598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.906709 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.915493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.915460 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.916301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916276 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.916655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916624 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.916744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916637 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.916744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916725 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.916835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916758 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.916835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916810 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.916971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.916952 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.917026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917007 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917039 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.917171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917105 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917177 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917217 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917283 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917392 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.917505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917470 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917507 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.917598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917541 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\") pod \"2995e2be-452e-40a3-9c38-164e786c2302\" (UID: \"2995e2be-452e-40a3-9c38-164e786c2302\") " Oct 30 19:35:22.917598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917571 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\") pod \"9a274584-6f3e-49a8-a802-97179d42a05e\" (UID: \"9a274584-6f3e-49a8-a802-97179d42a05e\") " Oct 30 19:35:22.918235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.917965 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.918235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.918191 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.918777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.918359 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.918777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.918431 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.919647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.919626 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:22.919786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.919772 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:22.919897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.919884 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:22.920175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.919988 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:22.920175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.920005 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:22.922752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.921598 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.922752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.921794 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.922752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.922238 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:22.922752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.922431 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.922752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.922567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.923552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.923529 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:22.924417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.924384 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:22.924847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.924787 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:22.924847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.924828 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "kube-api-access-zfpb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:22.925011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.924970 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "kube-api-access-9wgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:22.925546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.925512 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo" pod "2995e2be-452e-40a3-9c38-164e786c2302" (UID: "2995e2be-452e-40a3-9c38-164e786c2302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:22.932555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.931240 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:22.932759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.932725 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo" pod "9a274584-6f3e-49a8-a802-97179d42a05e" (UID: "9a274584-6f3e-49a8-a802-97179d42a05e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:22.999624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:22.999597 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:35:23.022242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022207 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022294 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022413 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022481 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022542 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022581 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022636 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022818 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.022926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.022908 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\") pod \"ff60e5c7-be31-4317-bff1-44e17babd616\" (UID: \"ff60e5c7-be31-4317-bff1-44e17babd616\") " Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023666 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023695 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023709 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2995e2be-452e-40a3-9c38-164e786c2302-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023722 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgh2\" (UniqueName: \"kubernetes.io/projected/9a274584-6f3e-49a8-a802-97179d42a05e-kube-api-access-9wgh2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023718 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023737 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a274584-6f3e-49a8-a802-97179d42a05e-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023751 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023762 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a274584-6f3e-49a8-a802-97179d42a05e-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.023769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023776 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfpb8\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-kube-api-access-zfpb8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023789 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023802 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a274584-6f3e-49a8-a802-97179d42a05e-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023816 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2995e2be-452e-40a3-9c38-164e786c2302-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023829 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2995e2be-452e-40a3-9c38-164e786c2302-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023842 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2995e2be-452e-40a3-9c38-164e786c2302-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.023949 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.024007 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.024106 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.024424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.024255 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:23.026533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.026483 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.027179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.027154 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:23.028419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.028375 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "kube-api-access-rdlvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:23.029012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.028984 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo" pod "ff60e5c7-be31-4317-bff1-44e17babd616" (UID: "ff60e5c7-be31-4317-bff1-44e17babd616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:23.056845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.056803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.070917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.070883 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:35:23.125123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125069 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125195 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125270 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125308 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125333 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125503 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125535 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125558 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.125587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.125581 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\") pod \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\" (UID: \"7c27d140-38d9-44f1-8537-fb9b30b4aa54\") " Oct 30 19:35:23.126145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126381 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126403 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff60e5c7-be31-4317-bff1-44e17babd616-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126417 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdlvt\" (UniqueName: \"kubernetes.io/projected/ff60e5c7-be31-4317-bff1-44e17babd616-kube-api-access-rdlvt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126430 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126442 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126453 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126465 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126476 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126484 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff60e5c7-be31-4317-bff1-44e17babd616-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.126942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.126492 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff60e5c7-be31-4317-bff1-44e17babd616-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.127616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.127009 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.127616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.127125 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.127616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.127149 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.127616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.127178 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.128767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.128734 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:23.128767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.128741 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "kube-api-access-wqxks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:23.128923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.128877 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:23.129868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.129843 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy" pod "7c27d140-38d9-44f1-8537-fb9b30b4aa54" (UID: "7c27d140-38d9-44f1-8537-fb9b30b4aa54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:23.228692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228649 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxks\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-kube-api-access-wqxks\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.228692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228677 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.228692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228689 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.228692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228699 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.229067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228707 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.229067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228717 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.229067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228729 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c27d140-38d9-44f1-8537-fb9b30b4aa54-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.229067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.228741 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c27d140-38d9-44f1-8537-fb9b30b4aa54-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:23.318647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.318614 4990 generic.go:356] "Generic (PLEG): container finished" podID="2995e2be-452e-40a3-9c38-164e786c2302" containerID="7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73" exitCode=0 Oct 30 19:35:23.318831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.318667 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" Oct 30 19:35:23.318831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.318680 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" event={"ID":"2995e2be-452e-40a3-9c38-164e786c2302","Type":"ContainerDied","Data":"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73"} Oct 30 19:35:23.318831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.318740 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k" event={"ID":"2995e2be-452e-40a3-9c38-164e786c2302","Type":"ContainerDied","Data":"3f9546fa13709fd73b2d38112cd5f6de17f13db313db20d64d19a72f8b3f6b95"} Oct 30 19:35:23.318831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.318772 4990 scope.go:122] "RemoveContainer" containerID="7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73" Oct 30 19:35:23.320851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.320821 4990 generic.go:356] "Generic (PLEG): container finished" podID="9a274584-6f3e-49a8-a802-97179d42a05e" containerID="7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31" exitCode=0 Oct 30 19:35:23.321150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.320876 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" event={"ID":"9a274584-6f3e-49a8-a802-97179d42a05e","Type":"ContainerDied","Data":"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31"} Oct 30 19:35:23.321307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.320901 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" Oct 30 19:35:23.321524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.321495 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4" event={"ID":"9a274584-6f3e-49a8-a802-97179d42a05e","Type":"ContainerDied","Data":"b3902cea82ff68fcc6e625ed3bbd4258cad103fa27a501ed7e1f3489e0b57c4d"} Oct 30 19:35:23.327458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.323314 4990 generic.go:356] "Generic (PLEG): container finished" podID="ff60e5c7-be31-4317-bff1-44e17babd616" containerID="1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab" exitCode=0 Oct 30 19:35:23.327458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.323434 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" event={"ID":"ff60e5c7-be31-4317-bff1-44e17babd616","Type":"ContainerDied","Data":"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab"} Oct 30 19:35:23.327458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.323468 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" event={"ID":"ff60e5c7-be31-4317-bff1-44e17babd616","Type":"ContainerDied","Data":"e7910bcc90c77455c3b8881f4613137e9d04805cabb7065e0c94dd8500a91cec"} Oct 30 19:35:23.327458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.323528 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k" Oct 30 19:35:23.327781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.327721 4990 generic.go:356] "Generic (PLEG): container finished" podID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerID="da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340" exitCode=0 Oct 30 19:35:23.327781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.327770 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" event={"ID":"7c27d140-38d9-44f1-8537-fb9b30b4aa54","Type":"ContainerDied","Data":"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340"} Oct 30 19:35:23.327899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.327803 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" event={"ID":"7c27d140-38d9-44f1-8537-fb9b30b4aa54","Type":"ContainerDied","Data":"5c9dd2f84f86dced6258386575e37ea02f2a2173b67f533da3215fd583a660da"} Oct 30 19:35:23.327899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.327878 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh" Oct 30 19:35:23.335711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.335690 4990 scope.go:122] "RemoveContainer" containerID="7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73" Oct 30 19:35:23.336171 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:23.336047 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73\": container with ID starting with 7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73 not found: ID does not exist" containerID="7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73" Oct 30 19:35:23.336171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.336083 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73"} err="failed to get container status \"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73\": rpc error: code = NotFound desc = could not find container \"7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73\": container with ID starting with 7934ff195f2e6f824c6e0acdfcd15f2830982d8d5ccf3300f89a209ef382db73 not found: ID does not exist" Oct 30 19:35:23.336171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.336125 4990 scope.go:122] "RemoveContainer" containerID="7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31" Oct 30 19:35:23.352220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.352197 4990 scope.go:122] "RemoveContainer" containerID="7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31" Oct 30 19:35:23.352615 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:23.352573 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31\": container with ID starting with 7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31 not found: ID does not exist" containerID="7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31" Oct 30 19:35:23.352722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.352633 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31"} err="failed to get container status \"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31\": rpc error: code = NotFound desc = could not find container \"7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31\": container with ID starting with 7754b6d45c16e42cf6d9e05d68b62ab477fcd212443a39b44d4d56c6ed3d0b31 not found: ID does not exist" Oct 30 19:35:23.352722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.352669 4990 scope.go:122] "RemoveContainer" containerID="1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab" Oct 30 19:35:23.364139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.364078 4990 scope.go:122] "RemoveContainer" containerID="1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab" Oct 30 19:35:23.364458 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:23.364428 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab\": container with ID starting with 1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab not found: ID does not exist" containerID="1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab" Oct 30 19:35:23.364545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.364469 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab"} err="failed to get container status \"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab\": rpc error: code = NotFound desc = could not find container \"1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab\": container with ID starting with 1af0b3c812b4cf28e414811f0a6d1ec5674b81fa0a4af9a18d0837313fa5ebab not found: ID does not exist" Oct 30 19:35:23.364545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.364492 4990 scope.go:122] "RemoveContainer" containerID="da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340" Oct 30 19:35:23.377985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.377933 4990 scope.go:122] "RemoveContainer" containerID="da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340" Oct 30 19:35:23.378316 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:23.378295 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340\": container with ID starting with da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340 not found: ID does not exist" containerID="da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340" Oct 30 19:35:23.378414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.378322 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340"} err="failed to get container status \"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340\": rpc error: code = NotFound desc = could not find container \"da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340\": container with ID starting with da545bf61e932353fabce56a628348a10cfbd501c7c32e9bff2a4b01601ff340 not found: ID does not exist" Oct 30 19:35:23.419305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.419270 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:35:23.429248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.429211 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-nlf4k"] Oct 30 19:35:23.510814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.496117 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:35:23.517687 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.512525 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-fwb4k"] Oct 30 19:35:23.522117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.521450 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:35:23.527525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.527495 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-mx4dh"] Oct 30 19:35:23.552376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.552075 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:35:23.560587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.560551 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-9pwd4"] Oct 30 19:35:23.757522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.757436 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:35:23.758208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758188 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758208 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758236 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758241 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758258 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758263 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" Oct 30 19:35:23.758283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758282 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" Oct 30 19:35:23.758600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758290 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" Oct 30 19:35:23.758600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758416 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2995e2be-452e-40a3-9c38-164e786c2302" containerName="istio-proxy" Oct 30 19:35:23.758600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758431 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" containerName="istio-proxy" Oct 30 19:35:23.758600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758438 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" containerName="istio-proxy" Oct 30 19:35:23.758600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.758446 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" containerName="istio-proxy" Oct 30 19:35:23.764276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.764253 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.783714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.782080 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8e6b67-openshift-default-dockercfg-88w6p\"" Oct 30 19:35:23.790845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.790816 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:35:23.844181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844152 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844211 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844337 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844470 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844546 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844654 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844759 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844815 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk8k\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.844893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.844885 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.854652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.854625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.946825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.946776 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.946868 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947018 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947155 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947282 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947388 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947458 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk8k\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947513 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.947607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.947578 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.948450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.948426 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.948450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.948433 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.948722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.948695 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.949240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.949212 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.949372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.949316 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.952293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.952272 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.955406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.955380 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.978376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.978219 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk8k\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:23.978689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:23.978667 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\") pod \"e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:24.087370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.087337 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:24.177871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.176841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:24.403565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.403479 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2995e2be-452e-40a3-9c38-164e786c2302" path="/var/lib/kubelet/pods/2995e2be-452e-40a3-9c38-164e786c2302/volumes" Oct 30 19:35:24.404036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.404018 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c27d140-38d9-44f1-8537-fb9b30b4aa54" path="/var/lib/kubelet/pods/7c27d140-38d9-44f1-8537-fb9b30b4aa54/volumes" Oct 30 19:35:24.404497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.404482 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a274584-6f3e-49a8-a802-97179d42a05e" path="/var/lib/kubelet/pods/9a274584-6f3e-49a8-a802-97179d42a05e/volumes" Oct 30 19:35:24.405276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.405259 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff60e5c7-be31-4317-bff1-44e17babd616" path="/var/lib/kubelet/pods/ff60e5c7-be31-4317-bff1-44e17babd616/volumes" Oct 30 19:35:24.432923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.432899 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:35:24.434548 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:35:24.434522 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c0b71e_6645_49b2_a868_c26fee1799ec.slice/crio-f0083f974df5a073eb09f879c29f238ab838992a6d00391f30c3375db12e1eed WatchSource:0}: Error finding container f0083f974df5a073eb09f879c29f238ab838992a6d00391f30c3375db12e1eed: Status 404 returned error can't find the container with id f0083f974df5a073eb09f879c29f238ab838992a6d00391f30c3375db12e1eed Oct 30 19:35:24.437507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.437475 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:24.437571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.437552 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:24.437608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:24.437582 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:25.162300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:25.162262 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:25.358496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:25.358457 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerStarted","Data":"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1"} Oct 30 19:35:25.358706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:25.358524 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerStarted","Data":"f0083f974df5a073eb09f879c29f238ab838992a6d00391f30c3375db12e1eed"} Oct 30 19:35:25.416952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:25.416841 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podStartSLOduration=2.416825019 podStartE2EDuration="2.416825019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:25.41273048 +0000 UTC m=+1833.124255848" watchObservedRunningTime="2026-10-30 19:35:25.416825019 +0000 UTC m=+1833.128350384" Oct 30 19:35:26.088171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.088139 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:26.094359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.094332 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:26.109114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.109075 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:35:26.188261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.188224 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\") pod \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " Oct 30 19:35:26.188719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.188280 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\" (UniqueName: \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\") pod \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\" (UID: \"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7\") " Oct 30 19:35:26.191897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.191453 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume" pod "5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" (UID: "5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:26.191897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.191595 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27" pod "5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" (UID: "5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7"). InnerVolumeSpecName "kube-api-access-58f27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:26.290316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.290267 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:26.290316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.290311 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f27\" (UniqueName: \"kubernetes.io/projected/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7-kube-api-access-58f27\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:26.368356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368265 4990 generic.go:356] "Generic (PLEG): container finished" podID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerID="a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe" exitCode=0 Oct 30 19:35:26.368356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368314 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" Oct 30 19:35:26.368581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368353 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" event={"ID":"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7","Type":"ContainerDied","Data":"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe"} Oct 30 19:35:26.368581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368411 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt" event={"ID":"5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7","Type":"ContainerDied","Data":"3150c474d799802b3bf667d9c08b387d910a4b5ad5b3eb3eb23ad0b3081939ef"} Oct 30 19:35:26.368581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368441 4990 scope.go:122] "RemoveContainer" containerID="a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe" Oct 30 19:35:26.368907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.368875 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:26.369986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.369961 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:35:26.383143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.383121 4990 scope.go:122] "RemoveContainer" containerID="a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe" Oct 30 19:35:26.383512 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:26.383486 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe\": container with ID starting with a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe not found: ID does not exist" containerID="a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe" Oct 30 19:35:26.383590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.383531 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe"} err="failed to get container status \"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe\": rpc error: code = NotFound desc = could not find container \"a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe\": container with ID starting with a8e4c96b346dd5a5dc36c10bf7f3d873f6d136aea19caa1ce8c81b37845ccbfe not found: ID does not exist" Oct 30 19:35:26.471318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.471262 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:35:26.482856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.482809 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j9xzt"] Oct 30 19:35:26.814177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.814135 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:35:26.815033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.815009 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" Oct 30 19:35:26.815033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.815032 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" Oct 30 19:35:26.815256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.815202 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" containerName="payload-processing" Oct 30 19:35:26.820386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.820357 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:26.843668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.843631 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:35:26.845623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.845589 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b625eb-serving-cert\"" Oct 30 19:35:26.901557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.901521 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:26.901746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.901694 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fr6\" (UniqueName: \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:26.902002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.901973 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:26.902214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.902126 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:26.942559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:26.942502 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.004303 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.004428 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.004554 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fr6\" (UniqueName: \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.004761 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.009760 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:27.011245 4990 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-serving-cert: secret "maas-api-e2e-api-b-b625eb-serving-cert" not found Oct 30 19:35:27.013269 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:27.011320 4990 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls podName:e25cd6d7-218b-4e04-b496-ef7d22a31f36 nodeName:}" failed. No retries permitted until 2026-10-30 19:35:27.511298454 +0000 UTC m=+1835.222823812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls") pod "maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" (UID: "e25cd6d7-218b-4e04-b496-ef7d22a31f36") : secret "maas-api-e2e-api-b-b625eb-serving-cert" not found Oct 30 19:35:27.013789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.013729 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.045155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.041162 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fr6\" (UniqueName: \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.083723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.083634 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.234925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.234883 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.514725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.514638 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.517074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.517048 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") pod \"maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:27.744010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:27.743977 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:28.058514 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:35:28.058478 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25cd6d7_218b_4e04_b496_ef7d22a31f36.slice/crio-c350f0dcc9dacb5144ab34f620e0bfd2a92b77088808ca7bce84d8a1a0ff3e5b WatchSource:0}: Error finding container c350f0dcc9dacb5144ab34f620e0bfd2a92b77088808ca7bce84d8a1a0ff3e5b: Status 404 returned error can't find the container with id c350f0dcc9dacb5144ab34f620e0bfd2a92b77088808ca7bce84d8a1a0ff3e5b Oct 30 19:35:28.059988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.059520 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:35:28.061134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.060350 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:35:28.122227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.122172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.375058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.375033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.386357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.386327 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" event={"ID":"e25cd6d7-218b-4e04-b496-ef7d22a31f36","Type":"ContainerStarted","Data":"c350f0dcc9dacb5144ab34f620e0bfd2a92b77088808ca7bce84d8a1a0ff3e5b"} Oct 30 19:35:28.403679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:28.403651 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7" path="/var/lib/kubelet/pods/5e8b87da-0a6e-4adf-aa47-4edbda0fd4f7/volumes" Oct 30 19:35:29.397224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:29.397183 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" event={"ID":"e25cd6d7-218b-4e04-b496-ef7d22a31f36","Type":"ContainerStarted","Data":"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0"} Oct 30 19:35:29.397635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:29.397253 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:29.438793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:29.438739 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" podStartSLOduration=3.159176156 podStartE2EDuration="3.438724637s" totalImagesPullingTime="279.548481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:29.43798994 +0000 UTC m=+1837.149515305" watchObservedRunningTime="2026-10-30 19:35:29.438724637 +0000 UTC m=+1837.150250003" Oct 30 19:35:29.475426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:29.475367 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.635377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.635337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.693725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.693692 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:35:30.698876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.698853 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.717900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.717868 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:35:30.724725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.724697 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8e6b67\"" Oct 30 19:35:30.767661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.767611 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:35:30.770585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.770554 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.771183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.771164 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mlt\" (UniqueName: \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.778424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.778401 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:30.791353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.790629 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.791494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.791337 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:35:30.797529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.797501 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8e6b67-dockercfg-tvd8m\"" Oct 30 19:35:30.891236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.890075 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgzd\" (UniqueName: \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:30.891236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.890193 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:30.891236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.890407 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mlt\" (UniqueName: \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.891236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.890466 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.973117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.937285 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:30.979715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.979649 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mlt\" (UniqueName: \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\") pod \"payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:31.003115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.991890 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgzd\" (UniqueName: \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:31.003115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.991978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:31.003115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:30.993055 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:31.014118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.013379 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:31.051235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.051173 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgzd\" (UniqueName: \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\") pod \"payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:31.103657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.103620 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:31.511662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.511630 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:35:31.513651 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:35:31.513151 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363207af_a81b_462b_9d87_7ffbf6e9fd63.slice/crio-41c7a57f00482cf6501c664941b53d2f94b9bc125d7f049cc7c2e438992d614a WatchSource:0}: Error finding container 41c7a57f00482cf6501c664941b53d2f94b9bc125d7f049cc7c2e438992d614a: Status 404 returned error can't find the container with id 41c7a57f00482cf6501c664941b53d2f94b9bc125d7f049cc7c2e438992d614a Oct 30 19:35:31.548700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.548673 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:35:31.848103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.848068 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:35:31.852715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.852696 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:31.867722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.867684 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9c8a2a-serving-cert\"" Oct 30 19:35:31.874055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.874023 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:35:31.947468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.947434 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:31.947697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.947682 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:31.947762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.947707 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvb4\" (UniqueName: \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:31.947762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.947741 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:31.990164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:31.990126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.049896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.049828 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.050505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.050471 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.050648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.050521 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvb4\" (UniqueName: \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.050818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.050796 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.051021 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:32.050995 4990 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-serving-cert: secret "maas-api-e2e-inf-a-9c8a2a-serving-cert" not found Oct 30 19:35:32.051200 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:32.051074 4990 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls podName:7ea9cc02-0ed9-43d5-902b-702155afd366 nodeName:}" failed. No retries permitted until 2026-10-30 19:35:32.551052719 +0000 UTC m=+1840.262578063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls") pod "maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" (UID: "7ea9cc02-0ed9-43d5-902b-702155afd366") : secret "maas-api-e2e-inf-a-9c8a2a-serving-cert" not found Oct 30 19:35:32.051696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.051674 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.053068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.053020 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.070683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.070649 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvb4\" (UniqueName: \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.126222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.126121 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.271687 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.271651 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.565918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.565873 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" event={"ID":"363207af-a81b-462b-9d87-7ffbf6e9fd63","Type":"ContainerStarted","Data":"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33"} Oct 30 19:35:32.566147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.565981 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" event={"ID":"363207af-a81b-462b-9d87-7ffbf6e9fd63","Type":"ContainerStarted","Data":"41c7a57f00482cf6501c664941b53d2f94b9bc125d7f049cc7c2e438992d614a"} Oct 30 19:35:32.566147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.566055 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:32.568341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.568315 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.568468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.568357 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" event={"ID":"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa","Type":"ContainerStarted","Data":"5f6d787743d041f3fb2fa7855125f53c435da421cc5e69b147adac6494771cdf"} Oct 30 19:35:32.568641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.568608 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" event={"ID":"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa","Type":"ContainerStarted","Data":"8173f90fe8e74d838ea94a68de451663879a3ea389ecdb6efb11ac3c1b363581"} Oct 30 19:35:32.568759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.568649 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:32.571578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.571557 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:32.607656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.607599 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" podStartSLOduration=2.607583069 podStartE2EDuration="2.607583069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:32.607477808 +0000 UTC m=+1840.319003174" watchObservedRunningTime="2026-10-30 19:35:32.607583069 +0000 UTC m=+1840.319108438" Oct 30 19:35:32.656934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.656883 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" podStartSLOduration=2.656869163 podStartE2EDuration="2.656869163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:32.651491843 +0000 UTC m=+1840.363017209" watchObservedRunningTime="2026-10-30 19:35:32.656869163 +0000 UTC m=+1840.368394528" Oct 30 19:35:32.774441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:32.774389 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:33.121779 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:35:33.121734 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea9cc02_0ed9_43d5_902b_702155afd366.slice/crio-afd8bc4fae69cce07c7dadb70c6a173e24121094d60701557685b1b94c034331 WatchSource:0}: Error finding container afd8bc4fae69cce07c7dadb70c6a173e24121094d60701557685b1b94c034331: Status 404 returned error can't find the container with id afd8bc4fae69cce07c7dadb70c6a173e24121094d60701557685b1b94c034331 Oct 30 19:35:33.121779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:33.121763 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:35:33.581764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:33.581726 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" event={"ID":"7ea9cc02-0ed9-43d5-902b-702155afd366","Type":"ContainerStarted","Data":"5e73a14957bd83225c4acb8777f8742fae9c1ebc6e3c576a4b032f99797adfc5"} Oct 30 19:35:33.581968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:33.581788 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" event={"ID":"7ea9cc02-0ed9-43d5-902b-702155afd366","Type":"ContainerStarted","Data":"afd8bc4fae69cce07c7dadb70c6a173e24121094d60701557685b1b94c034331"} Oct 30 19:35:33.582138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:33.582112 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:33.628121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:33.628057 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" podStartSLOduration=2.317094927 podStartE2EDuration="2.628043297s" totalImagesPullingTime="310.94837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:33.625252909 +0000 UTC m=+1841.336778277" watchObservedRunningTime="2026-10-30 19:35:33.628043297 +0000 UTC m=+1841.339568712" Oct 30 19:35:34.249227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:34.249193 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:34.525317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:34.525229 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.099962 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:35.099923 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:35:35.411669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:35.411585 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:35:36.007116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:36.005899 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:37.858557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:37.858469 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:39.598147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:39.598072 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:35:39.688026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:39.687990 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.612898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:41.612859 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:43.361962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:43.361921 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:43.584846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:43.584814 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:35:43.585027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:43.584885 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:35:45.533441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:45.533385 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.552484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:46.552441 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:47.161879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.161846 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:35:47.212835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.212796 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:35:47.370622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.370582 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:47.526222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.526073 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:47.758196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.758158 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:47.924853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:47.924817 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:48.550495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:48.550450 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.368039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:49.368005 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:50.049517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:50.049480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:52.162729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.162672 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" podUID="10b00a77-69da-4d93-a365-57dc0b8a0d74" containerName="payload-processing" containerID="cri-o://49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7" gracePeriod=25 Oct 30 19:35:52.213801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.213736 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" podUID="030a0045-99a8-48e8-aba0-5a4af5b4838c" containerName="payload-pre-processing" containerID="cri-o://c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c" gracePeriod=25 Oct 30 19:35:52.255630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.255594 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:52.556572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.556547 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:35:52.638467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.638426 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:35:52.673127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.673086 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\" (UniqueName: \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\") pod \"10b00a77-69da-4d93-a365-57dc0b8a0d74\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " Oct 30 19:35:52.673316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.673144 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\") pod \"10b00a77-69da-4d93-a365-57dc0b8a0d74\" (UID: \"10b00a77-69da-4d93-a365-57dc0b8a0d74\") " Oct 30 19:35:52.673573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.673544 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume" pod "10b00a77-69da-4d93-a365-57dc0b8a0d74" (UID: "10b00a77-69da-4d93-a365-57dc0b8a0d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:52.673943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.673920 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b00a77-69da-4d93-a365-57dc0b8a0d74-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:52.676357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.675875 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8" pod "10b00a77-69da-4d93-a365-57dc0b8a0d74" (UID: "10b00a77-69da-4d93-a365-57dc0b8a0d74"). InnerVolumeSpecName "kube-api-access-zxnq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:52.768928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.768835 4990 generic.go:356] "Generic (PLEG): container finished" podID="10b00a77-69da-4d93-a365-57dc0b8a0d74" containerID="49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7" exitCode=0 Oct 30 19:35:52.768928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.768887 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" Oct 30 19:35:52.768928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.768910 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" event={"ID":"10b00a77-69da-4d93-a365-57dc0b8a0d74","Type":"ContainerDied","Data":"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7"} Oct 30 19:35:52.769221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.768974 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4" event={"ID":"10b00a77-69da-4d93-a365-57dc0b8a0d74","Type":"ContainerDied","Data":"0f90eac3b5062e327607f354bf0a170355544b5764572de8fc9b5ab3ef3c151d"} Oct 30 19:35:52.769221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.769004 4990 scope.go:122] "RemoveContainer" containerID="49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7" Oct 30 19:35:52.770663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.770638 4990 generic.go:356] "Generic (PLEG): container finished" podID="030a0045-99a8-48e8-aba0-5a4af5b4838c" containerID="c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c" exitCode=0 Oct 30 19:35:52.770770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.770709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" event={"ID":"030a0045-99a8-48e8-aba0-5a4af5b4838c","Type":"ContainerDied","Data":"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c"} Oct 30 19:35:52.770770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.770734 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" Oct 30 19:35:52.770770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.770750 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj" event={"ID":"030a0045-99a8-48e8-aba0-5a4af5b4838c","Type":"ContainerDied","Data":"7692b570c42336b9c02a3cf323218e3ab78fe35464689a2cce68270b94efd27b"} Oct 30 19:35:52.774793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.774768 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\" (UniqueName: \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\") pod \"030a0045-99a8-48e8-aba0-5a4af5b4838c\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " Oct 30 19:35:52.774921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.774840 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\") pod \"030a0045-99a8-48e8-aba0-5a4af5b4838c\" (UID: \"030a0045-99a8-48e8-aba0-5a4af5b4838c\") " Oct 30 19:35:52.775370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.775340 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume" pod "030a0045-99a8-48e8-aba0-5a4af5b4838c" (UID: "030a0045-99a8-48e8-aba0-5a4af5b4838c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:52.775903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.775879 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxnq8\" (UniqueName: \"kubernetes.io/projected/10b00a77-69da-4d93-a365-57dc0b8a0d74-kube-api-access-zxnq8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:52.776005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.775911 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030a0045-99a8-48e8-aba0-5a4af5b4838c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:52.777205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.777177 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct" pod "030a0045-99a8-48e8-aba0-5a4af5b4838c" (UID: "030a0045-99a8-48e8-aba0-5a4af5b4838c"). InnerVolumeSpecName "kube-api-access-vvkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:52.783306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.783289 4990 scope.go:122] "RemoveContainer" containerID="49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7" Oct 30 19:35:52.783691 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:52.783663 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7\": container with ID starting with 49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7 not found: ID does not exist" containerID="49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7" Oct 30 19:35:52.783777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.783702 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7"} err="failed to get container status \"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7\": rpc error: code = NotFound desc = could not find container \"49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7\": container with ID starting with 49a727fa5c689c7a3401c18018adf3ecbc9026d65dcecdaff8ff56de663bf0a7 not found: ID does not exist" Oct 30 19:35:52.783777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.783726 4990 scope.go:122] "RemoveContainer" containerID="c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c" Oct 30 19:35:52.794712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.794690 4990 scope.go:122] "RemoveContainer" containerID="c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c" Oct 30 19:35:52.795013 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:35:52.794989 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c\": container with ID starting with c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c not found: ID does not exist" containerID="c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c" Oct 30 19:35:52.795057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.795024 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c"} err="failed to get container status \"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c\": rpc error: code = NotFound desc = could not find container \"c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c\": container with ID starting with c6a463050a1e6135e1fcf92ab1fe4d15c9ea6b2ecc2ab41f8c87c7a6f4a05a2c not found: ID does not exist" Oct 30 19:35:52.823287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.823263 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:35:52.827514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.827492 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-891d6b-6cd74ddc58-92sv4"] Oct 30 19:35:52.877955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:52.877922 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkct\" (UniqueName: \"kubernetes.io/projected/030a0045-99a8-48e8-aba0-5a4af5b4838c-kube-api-access-vvkct\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:35:53.118514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:53.118376 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:35:53.123940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:53.123908 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-891d6b-c57b68cdc-r84hj"] Oct 30 19:35:53.432836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:53.432725 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:53.596789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:53.596747 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:54.321395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:54.320116 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:54.408363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:54.408320 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030a0045-99a8-48e8-aba0-5a4af5b4838c" path="/var/lib/kubelet/pods/030a0045-99a8-48e8-aba0-5a4af5b4838c/volumes" Oct 30 19:35:54.408885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:54.408862 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b00a77-69da-4d93-a365-57dc0b8a0d74" path="/var/lib/kubelet/pods/10b00a77-69da-4d93-a365-57dc0b8a0d74/volumes" Oct 30 19:35:55.245701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.245672 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:55.414439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.414380 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:55.534589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.534500 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:35:55.535924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.535880 4990 cpu_manager.go:414] "containerMap: removing container" podUID="030a0045-99a8-48e8-aba0-5a4af5b4838c" containerName="payload-pre-processing" Oct 30 19:35:55.535924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.535922 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030a0045-99a8-48e8-aba0-5a4af5b4838c" containerName="payload-pre-processing" Oct 30 19:35:55.536155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.535993 4990 cpu_manager.go:414] "containerMap: removing container" podUID="10b00a77-69da-4d93-a365-57dc0b8a0d74" containerName="payload-processing" Oct 30 19:35:55.536155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.536002 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b00a77-69da-4d93-a365-57dc0b8a0d74" containerName="payload-processing" Oct 30 19:35:55.536332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.536312 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b00a77-69da-4d93-a365-57dc0b8a0d74" containerName="payload-processing" Oct 30 19:35:55.536392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.536347 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="030a0045-99a8-48e8-aba0-5a4af5b4838c" containerName="payload-pre-processing" Oct 30 19:35:55.545717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.544999 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.551482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.546929 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:35:55.590199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.589457 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4cb13-dockercfg-65bmb\"" Oct 30 19:35:55.618648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.599877 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4cb13\"" Oct 30 19:35:55.696140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.695289 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bxm\" (UniqueName: \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.696140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.695463 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.789649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.789398 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 19:35:55.797720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.797688 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.804135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.802909 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bxm\" (UniqueName: \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.804135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.803142 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.806346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.805912 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.827218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.816907 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-287ea2-openshift-default-dockercfg-7kw4v\"" Oct 30 19:35:55.831416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.831392 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bxm\" (UniqueName: \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\") pod \"payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:55.841523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.841493 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 19:35:55.917648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.917614 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.917827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.917787 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.917901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.917829 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5gc\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.917964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.917940 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.918029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.918012 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.918141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.918127 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.918212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.918151 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.918316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.918299 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.918379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.918344 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:55.944353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:55.944330 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:56.019750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.019659 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.019750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.019729 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.019970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.019797 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.019966 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.019986 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020006 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5gc\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020054 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020128 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020193 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020227 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020465 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020606 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.020793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.020774 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.021141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.021080 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.023364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.023323 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.023754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.023728 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.047118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.046970 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5gc\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.047118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.047037 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.152524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.152492 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:56.279305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.279254 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:35:56.326402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.326368 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:56.470253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.470221 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 19:35:56.500110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.496590 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:56.511125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.500602 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:56.511125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.507221 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:35:56.919169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.918638 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 19:35:56.923001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.922901 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" event={"ID":"be0a3c69-7a34-4427-b69d-33b2f0d11e35","Type":"ContainerStarted","Data":"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500"} Oct 30 19:35:56.923733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.923292 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" event={"ID":"be0a3c69-7a34-4427-b69d-33b2f0d11e35","Type":"ContainerStarted","Data":"2b7a0f7493527871480412915f6ec3af332b02760301b3558f3848e4e2d32c0d"} Oct 30 19:35:56.930228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.930202 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" event={"ID":"bb382239-d204-48c6-ab45-9cd2f873d831","Type":"ContainerStarted","Data":"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f"} Oct 30 19:35:56.930435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.930413 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" event={"ID":"bb382239-d204-48c6-ab45-9cd2f873d831","Type":"ContainerStarted","Data":"b7aab319c023235bffbabdc356f3664a989f8a58e18d06fd9c6988acbe859cfc"} Oct 30 19:35:56.932965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.931442 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:35:56.972461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:56.972383 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podStartSLOduration=1.972365564 podStartE2EDuration="1.972365564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:56.963778914 +0000 UTC m=+1864.675304289" watchObservedRunningTime="2026-10-30 19:35:56.972365564 +0000 UTC m=+1864.683890929" Oct 30 19:35:57.004392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.001564 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" podStartSLOduration=2.001543606 podStartE2EDuration="2.001543606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:57.000358393 +0000 UTC m=+1864.711883785" watchObservedRunningTime="2026-10-30 19:35:57.001543606 +0000 UTC m=+1864.713068965" Oct 30 19:35:57.155784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.155754 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:35:57.157026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.156998 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.157181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.157151 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:35:57.157244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.157194 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:35:57.302994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.302928 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.437411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:57.437297 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.153770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:58.153734 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:35:58.154137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:58.153805 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:35:59.153684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:59.153653 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:35:59.154195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:59.153703 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:35:59.363354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:59.363294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.684727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:35:59.684693 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.153167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:00.153077 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:36:00.153376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:00.153178 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:36:01.148318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:01.148265 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.153241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:01.153175 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:36:01.153554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:01.153251 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:36:01.826510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:01.826467 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:02.153425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:02.153342 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:36:02.153425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:02.153398 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:36:03.153207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:03.153163 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:36:03.153459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:03.153273 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 19:36:03.601237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:03.601195 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:04.158557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:04.158519 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:36:04.159057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:04.159040 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:36:04.159796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:04.159772 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 19:36:04.960897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:04.960854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:05.099041 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:36:05.099002 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:36:05.123390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:05.123359 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:05.951267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:05.951233 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.041296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:06.041263 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27"] Oct 30 19:36:06.052503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:06.052471 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266bd7e1-6kp27"] Oct 30 19:36:06.105697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:06.105666 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.268230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:06.264588 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.407156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:06.407126 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ef018-0808-49b2-bc82-b830cae35c38" path="/var/lib/kubelet/pods/f58ef018-0808-49b2-bc82-b830cae35c38/volumes" Oct 30 19:36:07.623692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:07.623655 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:07.781301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:07.781241 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.442791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:08.442749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.639506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:08.639460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.950624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:08.950594 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:36:10.667645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:10.667602 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:12.465059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:12.465006 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:14.343660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:14.343616 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:16.028937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:16.028894 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.296534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:17.296483 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.165266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:18.165231 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.441880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:18.441775 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.821171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:18.821133 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:19.544295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:19.538893 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:19.968053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:19.968008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.114236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:20.114204 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.596471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:20.596372 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:20.847562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:20.847469 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:21.891262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:21.891215 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:23.690884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:23.690847 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.471514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:24.471475 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:24.624128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:24.624076 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:25.901181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:25.901135 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:36:25.915135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:25.915057 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:25.927862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:25.927819 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:36:25.934804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:25.934668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.066876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.066834 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.066892 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.066961 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067005 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhw8\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067200 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067316 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067341 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067439 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.067635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.067620 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.168820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.168723 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.168981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.168893 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.168981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.168965 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169013 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169039 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169072 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhw8\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169201 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169219 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169300 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169316 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169329 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169641 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169936 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.169978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.169946 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.171913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.171891 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.172070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.172049 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.195149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.195031 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhw8\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.196508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.196482 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.265745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.264432 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:26.494469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.494084 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.637225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.637192 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:36:26.642262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.642241 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.647254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.647226 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:26.655253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.655219 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19f646\"" Oct 30 19:36:26.658889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.658867 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:36:26.674732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.674702 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19f646-dockercfg-k7msk\"" Oct 30 19:36:26.735790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.735705 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.737905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.737871 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k66ng\" (UniqueName: \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.771113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.766725 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:36:26.868112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.855007 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.868592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.868554 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k66ng\" (UniqueName: \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.889137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.869804 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.918603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.917578 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:26.918603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.917677 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:26.918603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.917719 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:26.958125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.956739 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k66ng\" (UniqueName: \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\") pod \"payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:26.961446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:26.961321 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:27.318168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:27.318132 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" event={"ID":"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d","Type":"ContainerStarted","Data":"ae5bb4a7bcc637c0bdf5086128fe109a3fa9bb6e9e5bb5be1b5b0b5642ab69d0"} Oct 30 19:36:27.318361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:27.318190 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" event={"ID":"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d","Type":"ContainerStarted","Data":"b93575377fbbbcc7a01cefd7a717600bc995e71d8610c9b326af91609167a7de"} Oct 30 19:36:27.360985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:27.360928 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:36:27.362919 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:36:27.362895 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3522964f_5fe5_432b_a4ab_7b8e8756d435.slice/crio-546574d8b17213512021f8595e27be4130f01a8abb655e73c9b18b6009dd0f5a WatchSource:0}: Error finding container 546574d8b17213512021f8595e27be4130f01a8abb655e73c9b18b6009dd0f5a: Status 404 returned error can't find the container with id 546574d8b17213512021f8595e27be4130f01a8abb655e73c9b18b6009dd0f5a Oct 30 19:36:27.363583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:27.363533 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podStartSLOduration=2.363516859 podStartE2EDuration="2.363516859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:27.357221951 +0000 UTC m=+1895.068747317" watchObservedRunningTime="2026-10-30 19:36:27.363516859 +0000 UTC m=+1895.075042225" Oct 30 19:36:27.878168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:27.876705 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.273170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.269386 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:28.279271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.279234 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:28.279539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.279509 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:28.336701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.336053 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" event={"ID":"3522964f-5fe5-432b-a4ab-7b8e8756d435","Type":"ContainerStarted","Data":"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd"} Oct 30 19:36:28.336701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.336118 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" event={"ID":"3522964f-5fe5-432b-a4ab-7b8e8756d435","Type":"ContainerStarted","Data":"546574d8b17213512021f8595e27be4130f01a8abb655e73c9b18b6009dd0f5a"} Oct 30 19:36:28.336701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.336567 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:28.379216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:28.379153 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" podStartSLOduration=2.379132904 podStartE2EDuration="2.379132904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.378492601 +0000 UTC m=+1896.090017967" watchObservedRunningTime="2026-10-30 19:36:28.379132904 +0000 UTC m=+1896.090658359" Oct 30 19:36:29.286117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.273488 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:29.286117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.273550 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:29.430829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.430792 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:36:29.486143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.475162 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:36:29.648120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.648044 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:29.836115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:29.829233 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:30.117115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:30.116812 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:30.282188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:30.282152 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:30.282656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:30.282605 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:31.274117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:31.269438 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:31.274117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:31.269503 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:32.244117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:32.234678 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:32.287113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:32.286882 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:32.287113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:32.286946 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:33.281053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:33.280521 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 19:36:33.281053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:33.280588 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 19:36:34.032953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.032920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.184584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.184545 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.275296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.274249 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:34.275296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.274813 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:34.276344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.276321 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:36:34.423971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.423898 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:34.431261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.431177 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" podUID="c73bf729-2ded-4316-ac13-1222f4990fa7" containerName="payload-processing" containerID="cri-o://115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c" gracePeriod=25 Oct 30 19:36:34.476408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.476365 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" containerName="payload-pre-processing" containerID="cri-o://5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683" gracePeriod=25 Oct 30 19:36:34.540936 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:36:34.540889 4990 logging.go:55] [core] [Channel #2933 SubChannel #2934]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.148:50052", ServerName: "10.134.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.148:50052: connect: connection refused" Oct 30 19:36:34.912895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.912858 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:36:34.916634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:34.916610 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:36:35.094312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.092196 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\" (UniqueName: \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\") pod \"c73bf729-2ded-4316-ac13-1222f4990fa7\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " Oct 30 19:36:35.094312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.092300 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\" (UniqueName: \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\") pod \"9c0bea28-7338-4e78-a206-c2c791367b02\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " Oct 30 19:36:35.094312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.092401 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\") pod \"c73bf729-2ded-4316-ac13-1222f4990fa7\" (UID: \"c73bf729-2ded-4316-ac13-1222f4990fa7\") " Oct 30 19:36:35.094312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.092545 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\") pod \"9c0bea28-7338-4e78-a206-c2c791367b02\" (UID: \"9c0bea28-7338-4e78-a206-c2c791367b02\") " Oct 30 19:36:35.094312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.094265 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume" pod "c73bf729-2ded-4316-ac13-1222f4990fa7" (UID: "c73bf729-2ded-4316-ac13-1222f4990fa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:35.095251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.095211 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume" pod "9c0bea28-7338-4e78-a206-c2c791367b02" (UID: "9c0bea28-7338-4e78-a206-c2c791367b02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:35.097327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.097202 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4" pod "9c0bea28-7338-4e78-a206-c2c791367b02" (UID: "9c0bea28-7338-4e78-a206-c2c791367b02"). InnerVolumeSpecName "kube-api-access-g94b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:35.097917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.097890 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn" pod "c73bf729-2ded-4316-ac13-1222f4990fa7" (UID: "c73bf729-2ded-4316-ac13-1222f4990fa7"). InnerVolumeSpecName "kube-api-access-9vwrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:35.104850 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:36:35.104815 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:36:35.195341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.195301 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94b4\" (UniqueName: \"kubernetes.io/projected/9c0bea28-7338-4e78-a206-c2c791367b02-kube-api-access-g94b4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:36:35.195341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.195342 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73bf729-2ded-4316-ac13-1222f4990fa7-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:36:35.195608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.195361 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0bea28-7338-4e78-a206-c2c791367b02-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:36:35.195608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.195374 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwrn\" (UniqueName: \"kubernetes.io/projected/c73bf729-2ded-4316-ac13-1222f4990fa7-kube-api-access-9vwrn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:36:35.477344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.477305 4990 generic.go:356] "Generic (PLEG): container finished" podID="9c0bea28-7338-4e78-a206-c2c791367b02" containerID="5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683" exitCode=0 Oct 30 19:36:35.477529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.477358 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" Oct 30 19:36:35.477529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.477394 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" event={"ID":"9c0bea28-7338-4e78-a206-c2c791367b02","Type":"ContainerDied","Data":"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683"} Oct 30 19:36:35.477529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.477466 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" event={"ID":"9c0bea28-7338-4e78-a206-c2c791367b02","Type":"ContainerDied","Data":"966728ef2b30184ac4c3c1c7b320489ca5a06c4e08c995f7bd710075e6c62ce1"} Oct 30 19:36:35.477529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.477499 4990 scope.go:122] "RemoveContainer" containerID="5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683" Oct 30 19:36:35.483051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.483026 4990 generic.go:356] "Generic (PLEG): container finished" podID="c73bf729-2ded-4316-ac13-1222f4990fa7" containerID="115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c" exitCode=0 Oct 30 19:36:35.484725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.484700 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" Oct 30 19:36:35.485076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.485057 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" event={"ID":"c73bf729-2ded-4316-ac13-1222f4990fa7","Type":"ContainerDied","Data":"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c"} Oct 30 19:36:35.485251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.485233 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6" event={"ID":"c73bf729-2ded-4316-ac13-1222f4990fa7","Type":"ContainerDied","Data":"4ffeb5e3083c75e518fd8550b143124f4c41b6cdbc80c11538de79a01ffa786e"} Oct 30 19:36:35.497562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.497542 4990 scope.go:122] "RemoveContainer" containerID="5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683" Oct 30 19:36:35.498072 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:36:35.498044 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683\": container with ID starting with 5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683 not found: ID does not exist" containerID="5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683" Oct 30 19:36:35.499360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.498140 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683"} err="failed to get container status \"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683\": rpc error: code = NotFound desc = could not find container \"5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683\": container with ID starting with 5abf6221453dc3ef4c2c3e911b93d9e5e6c4d059b87a372c85e471857d30a683 not found: ID does not exist" Oct 30 19:36:35.499360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.498166 4990 scope.go:122] "RemoveContainer" containerID="115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c" Oct 30 19:36:35.512962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.512938 4990 scope.go:122] "RemoveContainer" containerID="115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c" Oct 30 19:36:35.513307 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:36:35.513285 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c\": container with ID starting with 115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c not found: ID does not exist" containerID="115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c" Oct 30 19:36:35.513424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.513317 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c"} err="failed to get container status \"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c\": rpc error: code = NotFound desc = could not find container \"115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c\": container with ID starting with 115b5b1e8bd7aeb8f85102c0d58bd7e178893a998363543081fb60534d914c6c not found: ID does not exist" Oct 30 19:36:35.540565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.540441 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.148:50052\" within 1s: context deadline exceeded" Oct 30 19:36:35.554391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.554361 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:36:35.562333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.562299 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092bc0-74c8458bdc-w5d7m"] Oct 30 19:36:35.590454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.590420 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:36:35.597380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.597352 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092bc0-577cb9f7f4-nh2w6"] Oct 30 19:36:35.943811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:35.943773 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:36.406952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:36.406916 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" path="/var/lib/kubelet/pods/9c0bea28-7338-4e78-a206-c2c791367b02/volumes" Oct 30 19:36:36.407664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:36.407642 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73bf729-2ded-4316-ac13-1222f4990fa7" path="/var/lib/kubelet/pods/c73bf729-2ded-4316-ac13-1222f4990fa7/volumes" Oct 30 19:36:37.285786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.285749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:37.508615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.508527 4990 generic.go:356] "Generic (PLEG): container finished" podID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerID="2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1" exitCode=0 Oct 30 19:36:37.508996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.508607 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerDied","Data":"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1"} Oct 30 19:36:37.509231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.509211 4990 scope.go:122] "RemoveContainer" containerID="2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1" Oct 30 19:36:37.511247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.511215 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:37.511379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.511286 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:37.511379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.511322 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:36:37.790957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:37.790867 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:38.522177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:38.522140 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerStarted","Data":"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978"} Oct 30 19:36:39.101234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.101192 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:36:39.101451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.101391 4990 patch_prober.go:29] interesting pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 19:36:39.101525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.101459 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 19:36:39.304044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304011 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:36:39.304795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304773 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c73bf729-2ded-4316-ac13-1222f4990fa7" containerName="payload-processing" Oct 30 19:36:39.304795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304794 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73bf729-2ded-4316-ac13-1222f4990fa7" containerName="payload-processing" Oct 30 19:36:39.304962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304838 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" containerName="payload-pre-processing" Oct 30 19:36:39.304962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304845 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" containerName="payload-pre-processing" Oct 30 19:36:39.305067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304968 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0bea28-7338-4e78-a206-c2c791367b02" containerName="payload-pre-processing" Oct 30 19:36:39.305067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.304980 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73bf729-2ded-4316-ac13-1222f4990fa7" containerName="payload-processing" Oct 30 19:36:39.310184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.308971 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.334828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.334801 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ccf8bd61-serving-cert\"" Oct 30 19:36:39.341168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.341132 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:36:39.353227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.353079 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:36:39.403116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.402974 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fqp\" (UniqueName: \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.403116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.403077 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.403328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.403155 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.403370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.403352 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.505778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.505740 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.505990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.505962 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fqp\" (UniqueName: \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.506082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.506063 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.506178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.506121 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.507578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.507545 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.510685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.510655 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.511483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.511454 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.528908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.528874 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fqp\" (UniqueName: \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.539847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.539804 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:39.629649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.628379 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:39.752173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:39.740669 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:40.097414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:40.097286 4990 patch_prober.go:29] interesting pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 19:36:40.097414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:40.097353 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 19:36:40.098072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:40.097986 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:40.101798 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:36:40.101761 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8afbfd4_3372_4977_afa0_863fd793ff0f.slice/crio-96ba1c3d08239ff9aeccab9e35ce0a0c5181bf56a6913d28a3c3ead240b10fda WatchSource:0}: Error finding container 96ba1c3d08239ff9aeccab9e35ce0a0c5181bf56a6913d28a3c3ead240b10fda: Status 404 returned error can't find the container with id 96ba1c3d08239ff9aeccab9e35ce0a0c5181bf56a6913d28a3c3ead240b10fda Oct 30 19:36:40.102147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:40.102125 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:36:40.557358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:40.557313 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" event={"ID":"a8afbfd4-3372-4977-afa0-863fd793ff0f","Type":"ContainerStarted","Data":"96ba1c3d08239ff9aeccab9e35ce0a0c5181bf56a6913d28a3c3ead240b10fda"} Oct 30 19:36:41.089556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.089515 4990 patch_prober.go:29] interesting pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.089866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.089832 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 19:36:41.351847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.351759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:41.378249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.378208 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:36:41.571244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.571205 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" event={"ID":"a8afbfd4-3372-4977-afa0-863fd793ff0f","Type":"ContainerStarted","Data":"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d"} Oct 30 19:36:41.572992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.572964 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:41.618741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:41.618624 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" podStartSLOduration=2.214807503 podStartE2EDuration="2.618605823s" totalImagesPullingTime="403.79832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:41.612462506 +0000 UTC m=+1909.323987873" watchObservedRunningTime="2026-10-30 19:36:41.618605823 +0000 UTC m=+1909.330131191" Oct 30 19:36:42.095117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:42.094522 4990 patch_prober.go:29] interesting pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 19:36:42.095117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:42.094584 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 19:36:43.089602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:43.089559 4990 patch_prober.go:29] interesting pod/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 19:36:43.090241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:43.090194 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 19:36:43.303045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:43.303006 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:44.093413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:44.093381 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:36:44.094619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:44.094595 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:36:44.759015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:44.758956 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:45.530703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:45.530670 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:45.673356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:45.673321 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:45.832562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:45.832526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:46.392377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:46.392339 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.029449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:48.029413 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4"] Oct 30 19:36:48.040696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:48.040658 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.054347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:48.054309 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48aa6575-fxzt4"] Oct 30 19:36:48.407752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:48.407711 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adc20a3-8b76-4ea9-88d7-ced3860b52f6" path="/var/lib/kubelet/pods/1adc20a3-8b76-4ea9-88d7-ced3860b52f6/volumes" Oct 30 19:36:48.616967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:48.616935 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:36:49.832346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:49.832232 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:51.773387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:51.773350 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:53.629155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:53.629114 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:54.758608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:54.757490 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.439643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:55.439548 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.540529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:55.540498 4990 scope.go:122] "RemoveContainer" containerID="a29a72099b88e7521e80541a71868d7620b4e3d687dadd13de33c2c04d987a2d" Oct 30 19:36:55.551992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:55.551969 4990 scope.go:122] "RemoveContainer" containerID="117e6ecd120422c7ece88583f676582335be7bf0c9685480b19bdb1f05c1f65c" Oct 30 19:36:56.249929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:56.249892 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:56.432114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:56.431952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.204852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:57.193436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:58.757232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:58.757200 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:59.819712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:36:59.819674 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:00.543797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:00.543762 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:02.174678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:02.174641 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:02.855656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:02.855619 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:03.024269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:03.024226 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:03.872237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:03.872206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:04.016940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:04.016875 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:04.160730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:04.160575 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:05.119602 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:05.119540 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:37:06.118136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:06.118050 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:08.415026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:08.414976 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:10.329112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:10.328041 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:12.151802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:12.151765 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.168321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:13.168266 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.311157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:13.311121 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:13.969374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:13.969331 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:14.115046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:14.115005 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:15.972997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:15.972963 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:18.023220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:18.023171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:19.835457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:19.835363 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.096157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:20.096030 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.790342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:20.790302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:21.405122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:21.401805 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:21.827696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:21.827660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:22.113727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:22.113635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:23.949881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:23.949825 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.373981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.373947 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:37:24.421510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.421478 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:37:24.576063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.576028 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.646178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.646064 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:37:24.652230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.652203 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.691574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.666413 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8e6b67\"/\"test-model-8e6b67-kserve-self-signed-certs\"" Oct 30 19:37:24.691574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.666689 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8e6b67\"/\"kube-root-ca.crt\"" Oct 30 19:37:24.691574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.666804 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8e6b67\"/\"default-dockercfg-vjctm\"" Oct 30 19:37:24.691574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.666877 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8e6b67\"/\"openshift-service-ca.crt\"" Oct 30 19:37:24.691574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.673737 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:37:24.750202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.750164 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.764451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.764416 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp8mb\" (UniqueName: \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.764674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.764626 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.764767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.764741 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.764923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.764907 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.764984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.764953 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.765054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.765038 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.866997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.866955 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867057 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867255 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867295 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867408 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867448 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867463 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867544 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp8mb\" (UniqueName: \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.867724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.867706 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.870164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.870077 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.870291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.870186 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.884314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.884283 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp8mb\" (UniqueName: \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\") pod \"test-model-8e6b67-kserve-564bf858f9-4jpp9\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:24.891456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.891416 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:24.973676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:24.973589 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:25.442862 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:37:25.442810 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d57598a_9ec8_48e1_8b02_9a956dcb0341.slice/crio-afe4e7df566816fd292c3dccc2239ea09620bae963e87cf74becc2664d7cce21 WatchSource:0}: Error finding container afe4e7df566816fd292c3dccc2239ea09620bae963e87cf74becc2664d7cce21: Status 404 returned error can't find the container with id afe4e7df566816fd292c3dccc2239ea09620bae963e87cf74becc2664d7cce21 Oct 30 19:37:25.443586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:25.443513 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:37:25.931029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:25.930949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.061705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.061660 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" event={"ID":"3d57598a-9ec8-48e1-8b02-9a956dcb0341","Type":"ContainerStarted","Data":"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783"} Oct 30 19:37:26.062277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.061717 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" event={"ID":"3d57598a-9ec8-48e1-8b02-9a956dcb0341","Type":"ContainerStarted","Data":"afe4e7df566816fd292c3dccc2239ea09620bae963e87cf74becc2664d7cce21"} Oct 30 19:37:26.094292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.094252 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.102977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.102919 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" podStartSLOduration=2.102900677 podStartE2EDuration="2.102900677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:26.098172662 +0000 UTC m=+1953.809698028" watchObservedRunningTime="2026-10-30 19:37:26.102900677 +0000 UTC m=+1953.814426044" Oct 30 19:37:26.625186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.622949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:26.985170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:26.981170 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:28.503584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:28.503547 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.070053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.070019 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.263657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.263618 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.374881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.374741 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" podUID="0790539f-a33a-4715-b04a-cc82ac4caa11" containerName="payload-processing" containerID="cri-o://9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72" gracePeriod=25 Oct 30 19:37:29.422590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.422551 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" podUID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" containerName="payload-pre-processing" containerID="cri-o://27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248" gracePeriod=25 Oct 30 19:37:29.552870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.552781 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.830739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.829333 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:29.859438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.859315 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:37:29.869267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.869236 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:37:29.956205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.956168 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:37:29.957440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957411 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0790539f-a33a-4715-b04a-cc82ac4caa11" containerName="payload-processing" Oct 30 19:37:29.957440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957440 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0790539f-a33a-4715-b04a-cc82ac4caa11" containerName="payload-processing" Oct 30 19:37:29.957633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957475 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" containerName="payload-pre-processing" Oct 30 19:37:29.957633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957483 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" containerName="payload-pre-processing" Oct 30 19:37:29.957700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957659 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" containerName="payload-pre-processing" Oct 30 19:37:29.957700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.957690 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0790539f-a33a-4715-b04a-cc82ac4caa11" containerName="payload-processing" Oct 30 19:37:29.962241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.962217 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:29.976230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.975718 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a7c3b0\"" Oct 30 19:37:29.977485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.976519 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a7c3b0-dockercfg-rkpr2\"" Oct 30 19:37:29.977485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.976686 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:37:29.979638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.978245 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\") pod \"0790539f-a33a-4715-b04a-cc82ac4caa11\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " Oct 30 19:37:29.979638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.978299 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\" (UniqueName: \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\") pod \"0790539f-a33a-4715-b04a-cc82ac4caa11\" (UID: \"0790539f-a33a-4715-b04a-cc82ac4caa11\") " Oct 30 19:37:29.979638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.978463 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\" (UniqueName: \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\") pod \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " Oct 30 19:37:29.979638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.978533 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\") pod \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\" (UID: \"6a84b5e6-1a10-4cae-86f7-e3cf589c147f\") " Oct 30 19:37:29.980648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.980303 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume" pod "0790539f-a33a-4715-b04a-cc82ac4caa11" (UID: "0790539f-a33a-4715-b04a-cc82ac4caa11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:29.980982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.980930 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume" pod "6a84b5e6-1a10-4cae-86f7-e3cf589c147f" (UID: "6a84b5e6-1a10-4cae-86f7-e3cf589c147f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:29.985724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.985697 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl" pod "0790539f-a33a-4715-b04a-cc82ac4caa11" (UID: "0790539f-a33a-4715-b04a-cc82ac4caa11"). InnerVolumeSpecName "kube-api-access-jx5cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:29.986169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.986148 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn" pod "6a84b5e6-1a10-4cae-86f7-e3cf589c147f" (UID: "6a84b5e6-1a10-4cae-86f7-e3cf589c147f"). InnerVolumeSpecName "kube-api-access-sdksn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:29.994549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:29.994518 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.054118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.053255 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:37:30.078132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.071015 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.089872 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.090048 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqwz\" (UniqueName: \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.091292 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdksn\" (UniqueName: \"kubernetes.io/projected/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-kube-api-access-sdksn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.091318 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a84b5e6-1a10-4cae-86f7-e3cf589c147f-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.091335 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0790539f-a33a-4715-b04a-cc82ac4caa11-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:30.097859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.091348 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx5cl\" (UniqueName: \"kubernetes.io/projected/0790539f-a33a-4715-b04a-cc82ac4caa11-kube-api-access-jx5cl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:30.108325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.108283 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:37:30.158117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.141004 4990 generic.go:356] "Generic (PLEG): container finished" podID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" containerID="27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248" exitCode=0 Oct 30 19:37:30.158117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.141179 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" event={"ID":"6a84b5e6-1a10-4cae-86f7-e3cf589c147f","Type":"ContainerDied","Data":"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248"} Oct 30 19:37:30.158117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.141232 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" event={"ID":"6a84b5e6-1a10-4cae-86f7-e3cf589c147f","Type":"ContainerDied","Data":"2cf2a19c71de437fac7e5d01c9c8d1642e8961ec914b97c45a018076bc59aeb7"} Oct 30 19:37:30.158117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.141266 4990 scope.go:122] "RemoveContainer" containerID="27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248" Oct 30 19:37:30.158117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.141418 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk" Oct 30 19:37:30.169215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.169163 4990 generic.go:356] "Generic (PLEG): container finished" podID="0790539f-a33a-4715-b04a-cc82ac4caa11" containerID="9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72" exitCode=0 Oct 30 19:37:30.169614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.169583 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" event={"ID":"0790539f-a33a-4715-b04a-cc82ac4caa11","Type":"ContainerDied","Data":"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72"} Oct 30 19:37:30.169779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.169674 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" event={"ID":"0790539f-a33a-4715-b04a-cc82ac4caa11","Type":"ContainerDied","Data":"9761c06ac86c8479ad338a2ba7664d0a90238d20662ef8b581d46e2e882e4c0b"} Oct 30 19:37:30.169779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.169770 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9" Oct 30 19:37:30.195412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.195378 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqwz\" (UniqueName: \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.195835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.195816 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.196011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.195994 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.196428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.196407 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7k9\" (UniqueName: \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.198974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.198951 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.217997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.217891 4990 scope.go:122] "RemoveContainer" containerID="27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248" Oct 30 19:37:30.218670 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:30.218497 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248\": container with ID starting with 27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248 not found: ID does not exist" containerID="27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248" Oct 30 19:37:30.218670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.218538 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248"} err="failed to get container status \"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248\": rpc error: code = NotFound desc = could not find container \"27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248\": container with ID starting with 27c5a9c8f663e9e688869e13466dcb1f3255a126555bf3c61078429d9d1ed248 not found: ID does not exist" Oct 30 19:37:30.218670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.218566 4990 scope.go:122] "RemoveContainer" containerID="9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72" Oct 30 19:37:30.232132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.229558 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqwz\" (UniqueName: \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\") pod \"payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.267200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.267168 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:37:30.277231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.277193 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d9e82-84d68b46b6-p8sr9"] Oct 30 19:37:30.285583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.285548 4990 scope.go:122] "RemoveContainer" containerID="9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72" Oct 30 19:37:30.286249 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:30.286222 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72\": container with ID starting with 9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72 not found: ID does not exist" containerID="9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72" Oct 30 19:37:30.286473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.286445 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72"} err="failed to get container status \"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72\": rpc error: code = NotFound desc = could not find container \"9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72\": container with ID starting with 9a06f308ef58303bb1c7443338804d2dcdfc389bd0f51241f5e4180df4a9ce72 not found: ID does not exist" Oct 30 19:37:30.288458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.288428 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:30.300414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.300381 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.300552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.300513 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7k9\" (UniqueName: \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.301749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.301722 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.303812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.303786 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:37:30.308408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.308378 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d9e82-65c55ff5f-kfnpk"] Oct 30 19:37:30.318211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.318187 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7k9\" (UniqueName: \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\") pod \"payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.435686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.433827 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:30.457125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.456185 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0790539f-a33a-4715-b04a-cc82ac4caa11" path="/var/lib/kubelet/pods/0790539f-a33a-4715-b04a-cc82ac4caa11/volumes" Oct 30 19:37:30.457125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.457112 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a84b5e6-1a10-4cae-86f7-e3cf589c147f" path="/var/lib/kubelet/pods/6a84b5e6-1a10-4cae-86f7-e3cf589c147f/volumes" Oct 30 19:37:30.645161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.645120 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:37:30.798421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:30.798387 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:37:30.800590 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:37:30.800555 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c60a666_d1ed_40d1_86a9_6b07247e0617.slice/crio-de48fefe71a05b714d9a5dedf31b1433e63fd6b992e88aee32b437bd79e19272 WatchSource:0}: Error finding container de48fefe71a05b714d9a5dedf31b1433e63fd6b992e88aee32b437bd79e19272: Status 404 returned error can't find the container with id de48fefe71a05b714d9a5dedf31b1433e63fd6b992e88aee32b437bd79e19272 Oct 30 19:37:31.183586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.183548 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" event={"ID":"6b880856-0902-40a8-86a1-275db8694ad5","Type":"ContainerStarted","Data":"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956"} Oct 30 19:37:31.183776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.183609 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" event={"ID":"6b880856-0902-40a8-86a1-275db8694ad5","Type":"ContainerStarted","Data":"022f98f81ff116bc6b39b652fc78053f3c6236d086816f4fb8db656d31760af1"} Oct 30 19:37:31.183776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.183641 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:31.185258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.185230 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" event={"ID":"4c60a666-d1ed-40d1-86a9-6b07247e0617","Type":"ContainerStarted","Data":"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e"} Oct 30 19:37:31.185387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.185274 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" event={"ID":"4c60a666-d1ed-40d1-86a9-6b07247e0617","Type":"ContainerStarted","Data":"de48fefe71a05b714d9a5dedf31b1433e63fd6b992e88aee32b437bd79e19272"} Oct 30 19:37:31.185387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.185311 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:31.223916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.223866 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" podStartSLOduration=2.223842658 podStartE2EDuration="2.223842658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:31.222215193 +0000 UTC m=+1958.933740559" watchObservedRunningTime="2026-10-30 19:37:31.223842658 +0000 UTC m=+1958.935368026" Oct 30 19:37:31.265934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.265883 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" podStartSLOduration=1.265870697 podStartE2EDuration="1.265870697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:31.260927521 +0000 UTC m=+1958.972452888" watchObservedRunningTime="2026-10-30 19:37:31.265870697 +0000 UTC m=+1958.977396063" Oct 30 19:37:31.595335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:31.595298 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.449141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:33.449072 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:34.974704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:34.974657 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:34.987670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:34.987638 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:35.098220 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:35.098183 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:37:35.168336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:35.168293 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:35.237174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:35.237073 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:37:36.033360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:36.033317 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:36.282565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:36.282526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:36.356291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:36.356253 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:37:36.356546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:36.356516 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" containerID="cri-o://f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7" gracePeriod=30 Oct 30 19:37:36.448165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:36.448126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:37.347896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:37.347838 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:39.316166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:39.316126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:40.908829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:40.908794 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:41.675530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.675501 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:37:41.843750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843719 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.843935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843762 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.843935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843820 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.843935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843877 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843935 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.843978 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844021 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844028 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:41.844330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844174 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844291 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\") pod \"fa79589c-0eec-43ad-ab32-03644bd98b51\" (UID: \"fa79589c-0eec-43ad-ab32-03644bd98b51\") " Oct 30 19:37:41.844330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844307 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:41.844330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844306 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:41.844530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844378 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:41.844662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.844637 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:41.845237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.845210 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.845358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.845241 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.845358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.845259 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa79589c-0eec-43ad-ab32-03644bd98b51-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.845358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.845273 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.845358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.845288 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.846309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.846281 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:41.846407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.846373 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:41.846491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.846470 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:41.846529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.846505 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw" pod "fa79589c-0eec-43ad-ab32-03644bd98b51" (UID: "fa79589c-0eec-43ad-ab32-03644bd98b51"). InnerVolumeSpecName "kube-api-access-cgxnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:41.946713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.946623 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.946713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.946655 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.946713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.946667 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa79589c-0eec-43ad-ab32-03644bd98b51-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:41.946713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:41.946679 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxnw\" (UniqueName: \"kubernetes.io/projected/fa79589c-0eec-43ad-ab32-03644bd98b51-kube-api-access-cgxnw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:42.197028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.196936 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:37:42.197028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.197021 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:37:42.298509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.298471 4990 generic.go:356] "Generic (PLEG): container finished" podID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerID="f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7" exitCode=0 Oct 30 19:37:42.298953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.298548 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" Oct 30 19:37:42.300120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.298569 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerDied","Data":"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7"} Oct 30 19:37:42.300245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.300157 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f" event={"ID":"fa79589c-0eec-43ad-ab32-03644bd98b51","Type":"ContainerDied","Data":"e3bd77e4e5ac3bf32d81bfc11264180c614dcbde331a2937979b0928df03fc39"} Oct 30 19:37:42.300245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.300194 4990 scope.go:122] "RemoveContainer" containerID="f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7" Oct 30 19:37:42.319008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.318988 4990 scope.go:122] "RemoveContainer" containerID="86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805" Oct 30 19:37:42.329998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.329973 4990 scope.go:122] "RemoveContainer" containerID="f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7" Oct 30 19:37:42.330326 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:42.330301 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7\": container with ID starting with f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7 not found: ID does not exist" containerID="f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7" Oct 30 19:37:42.330447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.330333 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7"} err="failed to get container status \"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7\": rpc error: code = NotFound desc = could not find container \"f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7\": container with ID starting with f13715df990f08763886780a8f202becbee505c6ce4690c078ee948bdc108ac7 not found: ID does not exist" Oct 30 19:37:42.330447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.330351 4990 scope.go:122] "RemoveContainer" containerID="86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805" Oct 30 19:37:42.330668 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:42.330640 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805\": container with ID starting with 86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805 not found: ID does not exist" containerID="86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805" Oct 30 19:37:42.330714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.330680 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805"} err="failed to get container status \"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805\": rpc error: code = NotFound desc = could not find container \"86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805\": container with ID starting with 86fc9b69b12522ca8c0834276fd6f4e6dc9bd2041d43968690e449b23a880805 not found: ID does not exist" Oct 30 19:37:42.353204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.353177 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:37:42.362366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.362327 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-092bc0-openshift-default-6dd9ddd9b9-c6j9f"] Oct 30 19:37:42.411727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.411694 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" path="/var/lib/kubelet/pods/fa79589c-0eec-43ad-ab32-03644bd98b51/volumes" Oct 30 19:37:42.724161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:42.712021 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:44.407071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:44.406181 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:46.183834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:46.183717 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:47.419482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:47.419445 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:37:47.448847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:47.448815 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:37:47.603943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:47.603854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:47.793516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:47.793398 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:48.086910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.086716 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:48.975594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.975542 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9244-dd08be"] Oct 30 19:37:48.976640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976618 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.976727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976641 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.976727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976653 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.976727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976658 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.976895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976768 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.976895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.976775 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa79589c-0eec-43ad-ab32-03644bd98b51" containerName="istio-proxy" Oct 30 19:37:48.980629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:48.980612 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:49.009285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.009249 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9244-dd08be"] Oct 30 19:37:49.011463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.011437 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:37:49.086201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.086161 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfwc\" (UniqueName: \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\") pod \"test-curl-9244-dd08be\" (UID: \"42868458-a512-4ae8-a979-25a3e621758b\") " pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:49.187765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.187721 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfwc\" (UniqueName: \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\") pod \"test-curl-9244-dd08be\" (UID: \"42868458-a512-4ae8-a979-25a3e621758b\") " pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:49.213686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.213640 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfwc\" (UniqueName: \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\") pod \"test-curl-9244-dd08be\" (UID: \"42868458-a512-4ae8-a979-25a3e621758b\") " pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:49.295220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.295130 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:49.608915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:49.608886 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9244-dd08be"] Oct 30 19:37:49.611990 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:37:49.611954 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42868458_a512_4ae8_a979_25a3e621758b.slice/crio-33ee78dbab9fe2291617cbe00d92423e2a04f807512f8d1945ca2cde7a5fbffa WatchSource:0}: Error finding container 33ee78dbab9fe2291617cbe00d92423e2a04f807512f8d1945ca2cde7a5fbffa: Status 404 returned error can't find the container with id 33ee78dbab9fe2291617cbe00d92423e2a04f807512f8d1945ca2cde7a5fbffa Oct 30 19:37:50.085115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:50.079664 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:50.407301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:50.407210 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9244-dd08be" event={"ID":"42868458-a512-4ae8-a979-25a3e621758b","Type":"ContainerStarted","Data":"0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631"} Oct 30 19:37:50.407301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:50.407276 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9244-dd08be" event={"ID":"42868458-a512-4ae8-a979-25a3e621758b","Type":"ContainerStarted","Data":"33ee78dbab9fe2291617cbe00d92423e2a04f807512f8d1945ca2cde7a5fbffa"} Oct 30 19:37:50.432894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:50.432830 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9244-dd08be" podStartSLOduration=2.432812658 podStartE2EDuration="2.432812658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:50.428772231 +0000 UTC m=+1978.140297601" watchObservedRunningTime="2026-10-30 19:37:50.432812658 +0000 UTC m=+1978.144338025" Oct 30 19:37:51.199390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:51.199299 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:51.307725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:51.307684 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9244-dd08be"] Oct 30 19:37:51.312810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:51.312782 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9244-dd08be"] Oct 30 19:37:51.989137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:51.986721 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:52.411171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.411085 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9244-dd08be" podUID="42868458-a512-4ae8-a979-25a3e621758b" containerName="test-curl-9244-dd08be" containerID="cri-o://0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631" gracePeriod=2 Oct 30 19:37:52.419938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.419908 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" podUID="7f666317-1012-4e64-9078-0301067d1603" containerName="payload-processing" containerID="cri-o://368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9" gracePeriod=25 Oct 30 19:37:52.449115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.449072 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" podUID="37a2ac32-e3ea-434f-b883-39cb87d74a86" containerName="payload-pre-processing" containerID="cri-o://50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b" gracePeriod=25 Oct 30 19:37:52.766772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.766743 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:37:52.815278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.815248 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:37:52.866200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.866165 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\" (UniqueName: \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\") pod \"7f666317-1012-4e64-9078-0301067d1603\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " Oct 30 19:37:52.866406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.866228 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\") pod \"7f666317-1012-4e64-9078-0301067d1603\" (UID: \"7f666317-1012-4e64-9078-0301067d1603\") " Oct 30 19:37:52.866406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.866321 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\" (UniqueName: \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\") pod \"37a2ac32-e3ea-434f-b883-39cb87d74a86\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " Oct 30 19:37:52.866545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.866470 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\") pod \"37a2ac32-e3ea-434f-b883-39cb87d74a86\" (UID: \"37a2ac32-e3ea-434f-b883-39cb87d74a86\") " Oct 30 19:37:52.867676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.867643 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume" pod "37a2ac32-e3ea-434f-b883-39cb87d74a86" (UID: "37a2ac32-e3ea-434f-b883-39cb87d74a86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:52.867816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.867679 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume" pod "7f666317-1012-4e64-9078-0301067d1603" (UID: "7f666317-1012-4e64-9078-0301067d1603"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:52.879929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.879897 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs" pod "37a2ac32-e3ea-434f-b883-39cb87d74a86" (UID: "37a2ac32-e3ea-434f-b883-39cb87d74a86"). InnerVolumeSpecName "kube-api-access-sttbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:52.880066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.880007 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm" pod "7f666317-1012-4e64-9078-0301067d1603" (UID: "7f666317-1012-4e64-9078-0301067d1603"). InnerVolumeSpecName "kube-api-access-nk6zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:52.968523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.968439 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6zm\" (UniqueName: \"kubernetes.io/projected/7f666317-1012-4e64-9078-0301067d1603-kube-api-access-nk6zm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:52.968523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.968470 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f666317-1012-4e64-9078-0301067d1603-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:52.968523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.968499 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttbs\" (UniqueName: \"kubernetes.io/projected/37a2ac32-e3ea-434f-b883-39cb87d74a86-kube-api-access-sttbs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:52.968523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:52.968508 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a2ac32-e3ea-434f-b883-39cb87d74a86-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:53.423159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.423071 4990 generic.go:356] "Generic (PLEG): container finished" podID="37a2ac32-e3ea-434f-b883-39cb87d74a86" containerID="50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b" exitCode=0 Oct 30 19:37:53.423644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.423217 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" Oct 30 19:37:53.423644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.423241 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" event={"ID":"37a2ac32-e3ea-434f-b883-39cb87d74a86","Type":"ContainerDied","Data":"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b"} Oct 30 19:37:53.423644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.423302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m" event={"ID":"37a2ac32-e3ea-434f-b883-39cb87d74a86","Type":"ContainerDied","Data":"040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f"} Oct 30 19:37:53.423644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.423330 4990 scope.go:122] "RemoveContainer" containerID="50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b" Oct 30 19:37:53.425233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.425200 4990 generic.go:356] "Generic (PLEG): container finished" podID="7f666317-1012-4e64-9078-0301067d1603" containerID="368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9" exitCode=0 Oct 30 19:37:53.425379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.425250 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" Oct 30 19:37:53.425379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.425356 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" event={"ID":"7f666317-1012-4e64-9078-0301067d1603","Type":"ContainerDied","Data":"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9"} Oct 30 19:37:53.425500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.425390 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8" event={"ID":"7f666317-1012-4e64-9078-0301067d1603","Type":"ContainerDied","Data":"821a464c3a67ce2cb6b702dfcb50dfdde00c09a9d58013f9b79ce78167a64ab5"} Oct 30 19:37:53.450779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.450760 4990 scope.go:122] "RemoveContainer" containerID="50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b" Oct 30 19:37:53.451151 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:53.451075 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b\": container with ID starting with 50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b not found: ID does not exist" containerID="50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b" Oct 30 19:37:53.451271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.451183 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b"} err="failed to get container status \"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b\": rpc error: code = NotFound desc = could not find container \"50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b\": container with ID starting with 50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b not found: ID does not exist" Oct 30 19:37:53.451271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.451202 4990 scope.go:122] "RemoveContainer" containerID="368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9" Oct 30 19:37:53.461950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.461926 4990 scope.go:122] "RemoveContainer" containerID="368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9" Oct 30 19:37:53.462256 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:53.462234 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9\": container with ID starting with 368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9 not found: ID does not exist" containerID="368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9" Oct 30 19:37:53.462351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.462263 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9"} err="failed to get container status \"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9\": rpc error: code = NotFound desc = could not find container \"368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9\": container with ID starting with 368a9837053abf472ec37173911917423fae583ce0b0dacfdaab0efa35f763d9 not found: ID does not exist" Oct 30 19:37:53.483903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.483879 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:37:53.493479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.493424 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m"] Oct 30 19:37:53.516793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.516767 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:37:53.521590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.521569 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b625eb-6466fb5d76-nwhc8"] Oct 30 19:37:53.900143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:53.898480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:54.390949 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:54.390915 4990 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/0e2ea702ce66d6cd6b5f4963007dfbdacd177d95db747f44e65bd465843a643f/diff" to get inode usage: stat /var/lib/containers/storage/overlay/0e2ea702ce66d6cd6b5f4963007dfbdacd177d95db747f44e65bd465843a643f/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m_37a2ac32-e3ea-434f-b883-39cb87d74a86/payload-pre-processing/0.log" to get inode usage: stat /var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-b-b625eb-5f9b4dd9dc-7d57m_37a2ac32-e3ea-434f-b883-39cb87d74a86/payload-pre-processing/0.log: no such file or directory Oct 30 19:37:54.402652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.402613 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a2ac32-e3ea-434f-b883-39cb87d74a86" path="/var/lib/kubelet/pods/37a2ac32-e3ea-434f-b883-39cb87d74a86/volumes" Oct 30 19:37:54.403201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.403179 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f666317-1012-4e64-9078-0301067d1603" path="/var/lib/kubelet/pods/7f666317-1012-4e64-9078-0301067d1603/volumes" Oct 30 19:37:54.449660 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:54.449615 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f666317_1012_4e64_9078_0301067d1603.slice/crio-821a464c3a67ce2cb6b702dfcb50dfdde00c09a9d58013f9b79ce78167a64ab5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42868458_a512_4ae8_a979_25a3e621758b.slice/crio-0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2ac32_e3ea_434f_b883_39cb87d74a86.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f666317_1012_4e64_9078_0301067d1603.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42868458_a512_4ae8_a979_25a3e621758b.slice/crio-conmon-0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2ac32_e3ea_434f_b883_39cb87d74a86.slice/crio-50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2ac32_e3ea_434f_b883_39cb87d74a86.slice/crio-040de75c4df9a9bf35972326e7205e2acb2e57aa81b5eac297e9cc541f7ced0f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a2ac32_e3ea_434f_b883_39cb87d74a86.slice/crio-conmon-50069f7dfb35da47b8d89a8bbb557ee9650487b58bee0cc9efac9e98b3d4628b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42868458_a512_4ae8_a979_25a3e621758b.slice/crio-33ee78dbab9fe2291617cbe00d92423e2a04f807512f8d1945ca2cde7a5fbffa\": RecentStats: unable to find data in memory cache]" Oct 30 19:37:54.636049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.636021 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:54.697769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.697689 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\" (UniqueName: \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\") pod \"42868458-a512-4ae8-a979-25a3e621758b\" (UID: \"42868458-a512-4ae8-a979-25a3e621758b\") " Oct 30 19:37:54.699780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.699748 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc" pod "42868458-a512-4ae8-a979-25a3e621758b" (UID: "42868458-a512-4ae8-a979-25a3e621758b"). InnerVolumeSpecName "kube-api-access-8kfwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:54.800787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:54.800744 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfwc\" (UniqueName: \"kubernetes.io/projected/42868458-a512-4ae8-a979-25a3e621758b-kube-api-access-8kfwc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:37:55.448368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.448331 4990 generic.go:356] "Generic (PLEG): container finished" podID="42868458-a512-4ae8-a979-25a3e621758b" containerID="0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631" exitCode=137 Oct 30 19:37:55.448523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.448390 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:55.448523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.448431 4990 scope.go:122] "RemoveContainer" containerID="0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631" Oct 30 19:37:55.460500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.460474 4990 scope.go:122] "RemoveContainer" containerID="0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631" Oct 30 19:37:55.460873 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:55.460854 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631\": container with ID starting with 0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631 not found: ID does not exist" containerID="0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631" Oct 30 19:37:55.460919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.460890 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631"} err="failed to get container status \"0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631\": rpc error: code = NotFound desc = could not find container \"0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631\": container with ID starting with 0bd0313b02006fae49fa3cdfad635faeebaf92eb43c2580d302458504aee1631 not found: ID does not exist" Oct 30 19:37:55.471216 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:37:55.471186 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9244-dd08be\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="42868458-a512-4ae8-a979-25a3e621758b" pod="opendatahub/test-curl-9244-dd08be" Oct 30 19:37:55.514102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.514048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:55.612133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.612085 4990 scope.go:122] "RemoveContainer" containerID="1de69bbfcf7cdd6225d9b334da35dc3a98a559d365e8c8034a4766f13f7f399e" Oct 30 19:37:55.622837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.622814 4990 scope.go:122] "RemoveContainer" containerID="a932127a66cbada3a35994f88201e99fc28a733c7296ee1956446a480d579417" Oct 30 19:37:55.633537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:55.633518 4990 scope.go:122] "RemoveContainer" containerID="07ab362c8cf996facb28f03cf059e5e3a039fbe86a54337eb79e9438ac8a5168" Oct 30 19:37:56.406621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:56.406581 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42868458-a512-4ae8-a979-25a3e621758b" path="/var/lib/kubelet/pods/42868458-a512-4ae8-a979-25a3e621758b/volumes" Oct 30 19:37:57.144375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:57.144330 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:58.963120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:58.963064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:59.930883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:37:59.930850 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:00.457024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:00.456978 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:01.857000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:01.856963 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:38:01.857494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:01.857220 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" podUID="a8afbfd4-3372-4977-afa0-863fd793ff0f" containerName="maas-api" containerID="cri-o://e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d" gracePeriod=30 Oct 30 19:38:01.972792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:01.972749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.173284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.169840 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.176009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.175979 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:38:02.176846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176820 4990 cpu_manager.go:414] "containerMap: removing container" podUID="42868458-a512-4ae8-a979-25a3e621758b" containerName="test-curl-9244-dd08be" Oct 30 19:38:02.176846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176843 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42868458-a512-4ae8-a979-25a3e621758b" containerName="test-curl-9244-dd08be" Oct 30 19:38:02.177031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176889 4990 cpu_manager.go:414] "containerMap: removing container" podUID="37a2ac32-e3ea-434f-b883-39cb87d74a86" containerName="payload-pre-processing" Oct 30 19:38:02.177031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176897 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a2ac32-e3ea-434f-b883-39cb87d74a86" containerName="payload-pre-processing" Oct 30 19:38:02.177031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176947 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7f666317-1012-4e64-9078-0301067d1603" containerName="payload-processing" Oct 30 19:38:02.177031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.176955 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f666317-1012-4e64-9078-0301067d1603" containerName="payload-processing" Oct 30 19:38:02.177254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.177158 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f666317-1012-4e64-9078-0301067d1603" containerName="payload-processing" Oct 30 19:38:02.177254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.177182 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="42868458-a512-4ae8-a979-25a3e621758b" containerName="test-curl-9244-dd08be" Oct 30 19:38:02.177254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.177197 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a2ac32-e3ea-434f-b883-39cb87d74a86" containerName="payload-pre-processing" Oct 30 19:38:02.181103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.181072 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.198439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.197262 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4ab4a-openshift-default-dockercfg-hwh58\"" Oct 30 19:38:02.205478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.205451 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:38:02.237079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237041 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237413 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237682 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237732 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237832 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237899 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.237966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.237940 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.238128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.238038 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.238278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.238255 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhrz\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.242193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.242172 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:38:02.342115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.333588 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.348958 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349017 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349076 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349158 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349185 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349233 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349365 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhrz\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349395 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.349505 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.350199 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.350487 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.350726 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.351588 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.355111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.351833 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.429363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.426637 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.460117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.452417 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\") pod \"a8afbfd4-3372-4977-afa0-863fd793ff0f\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " Oct 30 19:38:02.460117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.452553 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\") pod \"a8afbfd4-3372-4977-afa0-863fd793ff0f\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " Oct 30 19:38:02.460117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.452919 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\" (UniqueName: \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\") pod \"a8afbfd4-3372-4977-afa0-863fd793ff0f\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " Oct 30 19:38:02.460117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.452951 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\") pod \"a8afbfd4-3372-4977-afa0-863fd793ff0f\" (UID: \"a8afbfd4-3372-4977-afa0-863fd793ff0f\") " Oct 30 19:38:02.464120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.460966 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca" pod "a8afbfd4-3372-4977-afa0-863fd793ff0f" (UID: "a8afbfd4-3372-4977-afa0-863fd793ff0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:02.482177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.470352 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8afbfd4-3372-4977-afa0-863fd793ff0f-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:02.482177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.479226 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls" pod "a8afbfd4-3372-4977-afa0-863fd793ff0f" (UID: "a8afbfd4-3372-4977-afa0-863fd793ff0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:02.507115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.484913 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhrz\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.507115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.486777 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.508939 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.518336 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp" pod "a8afbfd4-3372-4977-afa0-863fd793ff0f" (UID: "a8afbfd4-3372-4977-afa0-863fd793ff0f"). InnerVolumeSpecName "kube-api-access-z2fqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.518413 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs" pod "a8afbfd4-3372-4977-afa0-863fd793ff0f" (UID: "a8afbfd4-3372-4977-afa0-863fd793ff0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.522603 4990 generic.go:356] "Generic (PLEG): container finished" podID="a8afbfd4-3372-4977-afa0-863fd793ff0f" containerID="e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d" exitCode=0 Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.522795 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" event={"ID":"a8afbfd4-3372-4977-afa0-863fd793ff0f","Type":"ContainerDied","Data":"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d"} Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.522841 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" event={"ID":"a8afbfd4-3372-4977-afa0-863fd793ff0f","Type":"ContainerDied","Data":"96ba1c3d08239ff9aeccab9e35ce0a0c5181bf56a6913d28a3c3ead240b10fda"} Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.522863 4990 scope.go:122] "RemoveContainer" containerID="e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d" Oct 30 19:38:02.528110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.523005 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x" Oct 30 19:38:02.538177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.538144 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:02.544085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.544062 4990 scope.go:122] "RemoveContainer" containerID="e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d" Oct 30 19:38:02.544498 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:38:02.544471 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d\": container with ID starting with e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d not found: ID does not exist" containerID="e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d" Oct 30 19:38:02.544558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.544512 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d"} err="failed to get container status \"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d\": rpc error: code = NotFound desc = could not find container \"e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d\": container with ID starting with e2911cd1010a06fe463fb23c409421b357cc963c8517eeee2dd9437c0a66bc3d not found: ID does not exist" Oct 30 19:38:02.595150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.577640 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fqp\" (UniqueName: \"kubernetes.io/projected/a8afbfd4-3372-4977-afa0-863fd793ff0f-kube-api-access-z2fqp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:02.595150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.577674 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:02.595150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.577689 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8afbfd4-3372-4977-afa0-863fd793ff0f-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:02.595150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.586866 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:38:02.595150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.594033 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-4sw4x"] Oct 30 19:38:02.843843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.843800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:02.909118 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:38:02.905815 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f44ffe4_d341_4a70_a4a6_e37e8b270816.slice/crio-7c5674ee9983cdf93aa7afaf93d93dae192929b94a123863d8e658b1b6ee05ac WatchSource:0}: Error finding container 7c5674ee9983cdf93aa7afaf93d93dae192929b94a123863d8e658b1b6ee05ac: Status 404 returned error can't find the container with id 7c5674ee9983cdf93aa7afaf93d93dae192929b94a123863d8e658b1b6ee05ac Oct 30 19:38:02.909118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.909076 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:38:02.922161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.919856 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:38:02.922161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.919973 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:38:02.922161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:02.920014 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:38:03.533003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.532945 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" event={"ID":"5f44ffe4-d341-4a70-a4a6-e37e8b270816","Type":"ContainerStarted","Data":"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f"} Oct 30 19:38:03.533003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.533007 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" event={"ID":"5f44ffe4-d341-4a70-a4a6-e37e8b270816","Type":"ContainerStarted","Data":"7c5674ee9983cdf93aa7afaf93d93dae192929b94a123863d8e658b1b6ee05ac"} Oct 30 19:38:03.538249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.538224 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:03.582591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.582525 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podStartSLOduration=1.5825107219999999 podStartE2EDuration="1.582510722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:03.570103135 +0000 UTC m=+1991.281628496" watchObservedRunningTime="2026-10-30 19:38:03.582510722 +0000 UTC m=+1991.294036088" Oct 30 19:38:03.657276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.657238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:03.827988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:03.827949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.296208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.292650 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.407531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.407488 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8afbfd4-3372-4977-afa0-863fd793ff0f" path="/var/lib/kubelet/pods/a8afbfd4-3372-4977-afa0-863fd793ff0f/volumes" Oct 30 19:38:04.539808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.539776 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:38:04.539995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.539829 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:38:04.627916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.627878 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:38:04.628999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.628977 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a8afbfd4-3372-4977-afa0-863fd793ff0f" containerName="maas-api" Oct 30 19:38:04.629122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.629002 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8afbfd4-3372-4977-afa0-863fd793ff0f" containerName="maas-api" Oct 30 19:38:04.629279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.629262 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8afbfd4-3372-4977-afa0-863fd793ff0f" containerName="maas-api" Oct 30 19:38:04.633749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.633729 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.658945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.658916 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a7c3b0-serving-cert\"" Oct 30 19:38:04.673618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.665332 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:38:04.733514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.733482 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.733738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.733715 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llk2\" (UniqueName: \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.733895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.733874 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.733947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.733917 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.758380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.758348 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.835516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.835483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.835676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.835526 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.835676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.835652 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.835823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.835747 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llk2\" (UniqueName: \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.836392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.836364 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.838433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.838410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.838537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.838502 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.853959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.853935 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llk2\" (UniqueName: \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\") pod \"maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:04.919169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.918910 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:04.952152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:04.950629 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:05.116762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:05.116713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:05.148509 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:38:05.148474 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:38:05.539394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:05.539364 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:38:05.539798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:05.539415 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:38:05.609898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:05.609866 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:38:06.538648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.538613 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:38:06.538864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.538673 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:38:06.559722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.559690 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" event={"ID":"30682551-dd30-448a-930b-263880628b44","Type":"ContainerStarted","Data":"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28"} Oct 30 19:38:06.560160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.559749 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" event={"ID":"30682551-dd30-448a-930b-263880628b44","Type":"ContainerStarted","Data":"4ebe79638b4951f4a9d81daf3d4a3faa04f7ef98d326a38fb16fa5005c825ef5"} Oct 30 19:38:06.560160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.559790 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:06.618406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.618345 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" podStartSLOduration=2.194261172 podStartE2EDuration="2.618328046s" totalImagesPullingTime="424.066874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:06.614390864 +0000 UTC m=+1994.325916233" watchObservedRunningTime="2026-10-30 19:38:06.618328046 +0000 UTC m=+1994.329853415" Oct 30 19:38:06.911988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:06.911947 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:07.539209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:07.539172 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:38:07.539416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:07.539234 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:38:08.539479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:08.539436 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 19:38:08.539963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:08.539500 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 19:38:08.651432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:08.651390 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:09.331336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:09.331302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:09.740402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:09.740306 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:09.741364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:09.741193 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:09.742721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:09.742695 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:38:10.310656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:10.310620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:10.585586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:10.585500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:12.046117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:12.042694 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:12.594007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:12.593973 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:38:13.504708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:13.504669 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.212808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:14.212722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.392505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:14.392467 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.541881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:14.541778 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.036452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:15.036412 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.433915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:15.433873 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.579991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:15.579945 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:16.537430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:16.537378 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:18.254957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:18.254874 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:19.812002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:19.811213 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:21.052243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:21.052207 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:21.209137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:21.209084 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.026703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:22.025992 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.557518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:22.557479 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:22.871162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:22.871037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:23.080933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:23.080900 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:24.024551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:24.024510 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:24.673798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:24.673759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:24.845501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:24.845417 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:25.832323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:25.832283 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.511815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.511774 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.645585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.645553 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:38:26.650372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.650350 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.663789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.663758 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqjj\" (UniqueName: \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.663908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.663816 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.664818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.664795 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5d92fa\"" Oct 30 19:38:26.670838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.670815 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:38:26.688881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.688851 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.765305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.756804 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5d92fa-dockercfg-7b8s5\"" Oct 30 19:38:26.768549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.768514 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqjj\" (UniqueName: \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.768783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.768763 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.781117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.778893 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:38:26.845127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.834591 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:26.923147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.918727 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:26.923147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.918997 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:26.960436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.960388 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:26.964215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:26.964132 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:38:27.021017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.020917 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqjj\" (UniqueName: \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\") pod \"payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:27.021650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.021623 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:27.021983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.021966 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:27.026149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.023275 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:27.039942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.039916 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\") pod \"payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:27.265594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.265562 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:27.321295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.321258 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:27.578461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.577668 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:38:27.578461 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:38:27.577765 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a8917f_6024_4417_a379_02623868b1eb.slice/crio-983539631f5e59465a9eb917c44c7b18e05c20043d938bde56294e64ebc0e69a WatchSource:0}: Error finding container 983539631f5e59465a9eb917c44c7b18e05c20043d938bde56294e64ebc0e69a: Status 404 returned error can't find the container with id 983539631f5e59465a9eb917c44c7b18e05c20043d938bde56294e64ebc0e69a Oct 30 19:38:27.668309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.668261 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:38:27.669721 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:38:27.669686 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08de27d_a43f_4763_9c03_391b03f1a848.slice/crio-09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710 WatchSource:0}: Error finding container 09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710: Status 404 returned error can't find the container with id 09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710 Oct 30 19:38:27.956245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:27.955458 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:28.049261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.049222 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" event={"ID":"b08de27d-a43f-4763-9c03-391b03f1a848","Type":"ContainerStarted","Data":"9b520886d9a403b733e3eb80f6d44b6bccccc7e2dd918990f59ea27af63ad0d9"} Oct 30 19:38:28.049451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.049290 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" event={"ID":"b08de27d-a43f-4763-9c03-391b03f1a848","Type":"ContainerStarted","Data":"09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710"} Oct 30 19:38:28.049451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.049436 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:28.051012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.050983 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" event={"ID":"16a8917f-6024-4417-a379-02623868b1eb","Type":"ContainerStarted","Data":"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30"} Oct 30 19:38:28.051185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.051027 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" event={"ID":"16a8917f-6024-4417-a379-02623868b1eb","Type":"ContainerStarted","Data":"983539631f5e59465a9eb917c44c7b18e05c20043d938bde56294e64ebc0e69a"} Oct 30 19:38:28.051185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.051118 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:28.087453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.087407 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" podStartSLOduration=2.087393581 podStartE2EDuration="2.087393581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.084415312 +0000 UTC m=+2015.795940678" watchObservedRunningTime="2026-10-30 19:38:28.087393581 +0000 UTC m=+2015.798918998" Oct 30 19:38:28.123888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:28.123830 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" podStartSLOduration=2.123817194 podStartE2EDuration="2.123817194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.118350736 +0000 UTC m=+2015.829876103" watchObservedRunningTime="2026-10-30 19:38:28.123817194 +0000 UTC m=+2015.835342559" Oct 30 19:38:29.770156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:29.761691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:31.613254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:31.613218 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:33.334165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:33.334123 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:33.796709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:33.796618 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.405113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:34.378929 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:35.105830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:35.105243 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:35.115609 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:38:35.115572 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:38:35.307274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:35.307238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:35.777712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:35.777675 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:38.272951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:38.272911 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:38.597790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:38.597754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:38.758654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:38.758620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:39.069722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:39.069681 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:38:39.069887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:39.069796 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:38:40.243618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:40.243573 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:40.623408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:40.623374 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:38:40.623614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:40.623589 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" containerID="cri-o://9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382" gracePeriod=30 Oct 30 19:38:40.948063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:40.947967 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:41.388075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:41.388037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.386231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:42.386194 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.527603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:42.527562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:42.680011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:42.679924 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:44.353544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:44.353496 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:45.779573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:45.779534 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:45.997748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:45.997712 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:38:46.069893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.069853 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.070252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.070163 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.070370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.070320 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.070569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.070550 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.070656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.070641 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.070725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.070692 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.071164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.071143 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.073052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.073033 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.073304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.072976 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.073662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.073465 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.073768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.073750 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.074058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.074043 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.074640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.074623 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\") pod \"9d2dac39-9736-4673-b47f-52ddbd76c017\" (UID: \"9d2dac39-9736-4673-b47f-52ddbd76c017\") " Oct 30 19:38:46.075842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.075819 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.075974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.075961 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.076059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.076048 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.076376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.076325 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.076376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.074513 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:46.083962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.083629 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:46.086141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.085215 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.086141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.085313 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:46.086592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.086556 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28" pod "9d2dac39-9736-4673-b47f-52ddbd76c017" (UID: "9d2dac39-9736-4673-b47f-52ddbd76c017"). InnerVolumeSpecName "kube-api-access-lxj28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:46.178222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.178185 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d2dac39-9736-4673-b47f-52ddbd76c017-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.178222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.178218 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.178222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.178227 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.178582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.178239 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.178582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.178251 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj28\" (UniqueName: \"kubernetes.io/projected/9d2dac39-9736-4673-b47f-52ddbd76c017-kube-api-access-lxj28\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.328154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.326924 4990 generic.go:356] "Generic (PLEG): container finished" podID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerID="9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382" exitCode=0 Oct 30 19:38:46.328154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.327025 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" Oct 30 19:38:46.328154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.327045 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" event={"ID":"9d2dac39-9736-4673-b47f-52ddbd76c017","Type":"ContainerDied","Data":"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382"} Oct 30 19:38:46.328154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.327110 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq" event={"ID":"9d2dac39-9736-4673-b47f-52ddbd76c017","Type":"ContainerDied","Data":"e35a94217a98801e5d3331e2728213941e2e9f18d59cfea4964b45eec85eaf07"} Oct 30 19:38:46.328154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.327137 4990 scope.go:122] "RemoveContainer" containerID="9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382" Oct 30 19:38:46.352391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.352376 4990 scope.go:122] "RemoveContainer" containerID="9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382" Oct 30 19:38:46.352878 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:38:46.352850 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382\": container with ID starting with 9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382 not found: ID does not exist" containerID="9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382" Oct 30 19:38:46.352966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.352891 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382"} err="failed to get container status \"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382\": rpc error: code = NotFound desc = could not find container \"9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382\": container with ID starting with 9a2630c42b0e28b7ae1577e937b1bf76c21e650862b98f497c239c2f82cd2382 not found: ID does not exist" Oct 30 19:38:46.389788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.389739 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:38:46.440811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.440717 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5x2wq"] Oct 30 19:38:46.665710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:46.665673 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:47.442039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:47.441988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:47.617347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:47.617278 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:48.405715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:48.405681 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" path="/var/lib/kubelet/pods/9d2dac39-9736-4673-b47f-52ddbd76c017/volumes" Oct 30 19:38:48.815668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:48.815618 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:50.822342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:50.822305 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:52.758458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:52.758419 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:53.198734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:53.198694 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:53.370445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:53.370407 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:54.284250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.284207 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:38:54.285005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.284969 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" podUID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" containerName="maas-api" containerID="cri-o://693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0" gracePeriod=30 Oct 30 19:38:54.407638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.407602 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:54.722830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.722795 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:38:54.968775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.968659 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\" (UniqueName: \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\") pod \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " Oct 30 19:38:54.973578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.969214 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") pod \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " Oct 30 19:38:54.974002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.973976 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\") pod \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " Oct 30 19:38:54.977235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.977209 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\") pod \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\" (UID: \"e25cd6d7-218b-4e04-b496-ef7d22a31f36\") " Oct 30 19:38:54.977794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.977743 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca" pod "e25cd6d7-218b-4e04-b496-ef7d22a31f36" (UID: "e25cd6d7-218b-4e04-b496-ef7d22a31f36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:54.978957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.978786 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25cd6d7-218b-4e04-b496-ef7d22a31f36-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:54.980561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.980534 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls" pod "e25cd6d7-218b-4e04-b496-ef7d22a31f36" (UID: "e25cd6d7-218b-4e04-b496-ef7d22a31f36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:54.982349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:54.982323 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs" pod "e25cd6d7-218b-4e04-b496-ef7d22a31f36" (UID: "e25cd6d7-218b-4e04-b496-ef7d22a31f36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:55.007147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.007084 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6" pod "e25cd6d7-218b-4e04-b496-ef7d22a31f36" (UID: "e25cd6d7-218b-4e04-b496-ef7d22a31f36"). InnerVolumeSpecName "kube-api-access-l9fr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:55.080892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.080852 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fr6\" (UniqueName: \"kubernetes.io/projected/e25cd6d7-218b-4e04-b496-ef7d22a31f36-kube-api-access-l9fr6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:55.080892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.080885 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:55.080892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.080896 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25cd6d7-218b-4e04-b496-ef7d22a31f36-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:38:55.221361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.221268 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:55.411243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.411213 4990 generic.go:356] "Generic (PLEG): container finished" podID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" containerID="693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0" exitCode=0 Oct 30 19:38:55.411700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.411259 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" event={"ID":"e25cd6d7-218b-4e04-b496-ef7d22a31f36","Type":"ContainerDied","Data":"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0"} Oct 30 19:38:55.411700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.411264 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" Oct 30 19:38:55.411700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.411305 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6" event={"ID":"e25cd6d7-218b-4e04-b496-ef7d22a31f36","Type":"ContainerDied","Data":"c350f0dcc9dacb5144ab34f620e0bfd2a92b77088808ca7bce84d8a1a0ff3e5b"} Oct 30 19:38:55.411700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.411336 4990 scope.go:122] "RemoveContainer" containerID="693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0" Oct 30 19:38:55.423916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.423865 4990 scope.go:122] "RemoveContainer" containerID="693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0" Oct 30 19:38:55.424240 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:38:55.424213 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0\": container with ID starting with 693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0 not found: ID does not exist" containerID="693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0" Oct 30 19:38:55.424372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.424247 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0"} err="failed to get container status \"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0\": rpc error: code = NotFound desc = could not find container \"693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0\": container with ID starting with 693b32ff34f2da0f840987ea2f09d0b503ef3810ce8dbf7a5fe672dde97de3d0 not found: ID does not exist" Oct 30 19:38:55.465781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.465753 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:38:55.478490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.478419 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b625eb-864db5f7bf-crxj6"] Oct 30 19:38:55.580135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.580046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:55.871124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:55.871076 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:56.258485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:56.258451 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:56.409028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:56.408988 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" path="/var/lib/kubelet/pods/e25cd6d7-218b-4e04-b496-ef7d22a31f36/volumes" Oct 30 19:38:57.986617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:57.986579 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.293598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:59.293560 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:38:59.490804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:59.490757 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.640882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:59.640841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.843200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:38:59.843158 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:00.032763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:00.032633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:00.182919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:00.182885 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:00.494956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:00.494920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:01.849337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:01.849293 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:02.785430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:02.785394 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:39:02.785674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:02.785649 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" containerID="cri-o://2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2" gracePeriod=30 Oct 30 19:39:02.795601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:02.795572 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:39:02.795851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:02.795812 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" containerID="cri-o://ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147" gracePeriod=30 Oct 30 19:39:02.988775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:02.988738 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.167755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.167717 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.377285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.377244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.555722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.555673 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:39:03.556295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.556250 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" containerID="cri-o://5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea" gracePeriod=30 Oct 30 19:39:03.586662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.586587 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.873386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.873281 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:39:03.876588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876559 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" Oct 30 19:39:03.876588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876584 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" Oct 30 19:39:03.876827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876630 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" containerName="maas-api" Oct 30 19:39:03.876827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876635 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" containerName="maas-api" Oct 30 19:39:03.876827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876742 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25cd6d7-218b-4e04-b496-ef7d22a31f36" containerName="maas-api" Oct 30 19:39:03.876827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.876754 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2dac39-9736-4673-b47f-52ddbd76c017" containerName="istio-proxy" Oct 30 19:39:03.882223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.882194 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:03.929130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.915114 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:39:03.929863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:03.928419 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9c8a2a-openshift-default-dockercfg-qsxpt\"" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.015910 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016016 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016044 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016072 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016121 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016189 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016242 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016388 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qspq\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.016781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.016531 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118277 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118385 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118414 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118441 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118477 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118544 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118598 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118726 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118756 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qspq\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.118948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118904 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.119017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.118993 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.119077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.119019 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.119325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.119304 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.119637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.119615 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.121950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.121930 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.122482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.122462 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.138812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.138674 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.140643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.140295 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qspq\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.170451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.169240 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.259990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.259961 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:04.294522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.294487 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" podUID="bb382239-d204-48c6-ab45-9cd2f873d831" containerName="payload-processing" containerID="cri-o://77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f" gracePeriod=25 Oct 30 19:39:04.555969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.555911 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.583118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.580966 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:39:04.583118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.581894 4990 generic.go:356] "Generic (PLEG): container finished" podID="bb382239-d204-48c6-ab45-9cd2f873d831" containerID="77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f" exitCode=0 Oct 30 19:39:04.583118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.581928 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" event={"ID":"bb382239-d204-48c6-ab45-9cd2f873d831","Type":"ContainerDied","Data":"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f"} Oct 30 19:39:04.583118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.582009 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" event={"ID":"bb382239-d204-48c6-ab45-9cd2f873d831","Type":"ContainerDied","Data":"b7aab319c023235bffbabdc356f3664a989f8a58e18d06fd9c6988acbe859cfc"} Oct 30 19:39:04.583118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.582041 4990 scope.go:122] "RemoveContainer" containerID="77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f" Oct 30 19:39:04.601411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.600518 4990 scope.go:122] "RemoveContainer" containerID="77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f" Oct 30 19:39:04.601411 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:04.601311 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f\": container with ID starting with 77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f not found: ID does not exist" containerID="77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f" Oct 30 19:39:04.601411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.601353 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f"} err="failed to get container status \"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f\": rpc error: code = NotFound desc = could not find container \"77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f\": container with ID starting with 77929a80a3841c3a1fc2a5ef49815664d2ca66e3216fbfe21b71405e8287063f not found: ID does not exist" Oct 30 19:39:04.603038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.603013 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:39:04.612321 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:39:04.610544 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c58efb1_eea9_40e8_9951_5b8e8b7ba230.slice/crio-f196fe4670bda5dbd14420a75cff7d4a50a749c5787a4e872c0d31f97c080aad WatchSource:0}: Error finding container f196fe4670bda5dbd14420a75cff7d4a50a749c5787a4e872c0d31f97c080aad: Status 404 returned error can't find the container with id f196fe4670bda5dbd14420a75cff7d4a50a749c5787a4e872c0d31f97c080aad Oct 30 19:39:04.614571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.614529 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:04.614677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.614623 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:04.614677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.614668 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:04.725870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.725166 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.738540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.736306 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\" (UniqueName: \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\") pod \"bb382239-d204-48c6-ab45-9cd2f873d831\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " Oct 30 19:39:04.738540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.736672 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\") pod \"bb382239-d204-48c6-ab45-9cd2f873d831\" (UID: \"bb382239-d204-48c6-ab45-9cd2f873d831\") " Oct 30 19:39:04.742745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.742697 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume" pod "bb382239-d204-48c6-ab45-9cd2f873d831" (UID: "bb382239-d204-48c6-ab45-9cd2f873d831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:04.745471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.745429 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm" pod "bb382239-d204-48c6-ab45-9cd2f873d831" (UID: "bb382239-d204-48c6-ab45-9cd2f873d831"). InnerVolumeSpecName "kube-api-access-t9bxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:04.921115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.886309 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb382239-d204-48c6-ab45-9cd2f873d831-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:04.921115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.886353 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bxm\" (UniqueName: \"kubernetes.io/projected/bb382239-d204-48c6-ab45-9cd2f873d831-kube-api-access-t9bxm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:04.961649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:04.960699 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:05.108726 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:05.108680 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:39:05.613889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.613849 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" event={"ID":"8c58efb1-eea9-40e8-9951-5b8e8b7ba230","Type":"ContainerStarted","Data":"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466"} Oct 30 19:39:05.614129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.613940 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" event={"ID":"8c58efb1-eea9-40e8-9951-5b8e8b7ba230","Type":"ContainerStarted","Data":"f196fe4670bda5dbd14420a75cff7d4a50a749c5787a4e872c0d31f97c080aad"} Oct 30 19:39:05.615180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.615154 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph" Oct 30 19:39:05.663206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.663136 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podStartSLOduration=2.66311953 podStartE2EDuration="2.66311953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:05.660234848 +0000 UTC m=+2053.371760218" watchObservedRunningTime="2026-10-30 19:39:05.66311953 +0000 UTC m=+2053.374644898" Oct 30 19:39:05.708224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.708183 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:39:05.712683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:05.712648 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4cb13-7f86b775cb-dcgph"] Oct 30 19:39:06.136809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.136771 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:39:06.137601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.137584 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bb382239-d204-48c6-ab45-9cd2f873d831" containerName="payload-processing" Oct 30 19:39:06.137652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.137603 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb382239-d204-48c6-ab45-9cd2f873d831" containerName="payload-processing" Oct 30 19:39:06.137730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.137721 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb382239-d204-48c6-ab45-9cd2f873d831" containerName="payload-processing" Oct 30 19:39:06.141371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.141355 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.159400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.159376 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5d92fa-serving-cert\"" Oct 30 19:39:06.161784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.161758 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:39:06.216719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.216680 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.216921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.216724 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcxm\" (UniqueName: \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.217102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.217057 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.217270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.217246 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.261046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.261014 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:06.262570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.262541 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:06.262712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.262595 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:06.293144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.292184 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:06.318316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.318278 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.318534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.318384 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.318534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.318444 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.318534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.318468 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcxm\" (UniqueName: \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.319202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.319177 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.322072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.322047 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.322187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.322046 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.336217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.336194 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcxm\" (UniqueName: \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\") pod \"maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.437161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.434051 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb382239-d204-48c6-ab45-9cd2f873d831" path="/var/lib/kubelet/pods/bb382239-d204-48c6-ab45-9cd2f873d831/volumes" Oct 30 19:39:06.437161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.436343 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:06.455731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.455694 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:06.833409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:06.833375 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:39:07.261058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:07.260964 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:07.261058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:07.261036 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:07.648226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:07.648183 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" event={"ID":"f0029510-1083-4fec-935d-03d20bcdda8b","Type":"ContainerStarted","Data":"35ed134d8b54fc02ade0969c5593db03ce4bddee106471ca2beff30c40e2e6d9"} Oct 30 19:39:07.648404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:07.648251 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" event={"ID":"f0029510-1083-4fec-935d-03d20bcdda8b","Type":"ContainerStarted","Data":"d2bba09f8b65b4d1a3317fd7a411deaae97208a9d8acb6aa40063e8168495470"} Oct 30 19:39:07.648404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:07.648283 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:08.173114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.170285 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:39:08.232801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.232689 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" podStartSLOduration=1.8266758140000001 podStartE2EDuration="2.232668737s" totalImagesPullingTime="405.992923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:07.698726745 +0000 UTC m=+2055.410252112" watchObservedRunningTime="2026-10-30 19:39:08.232668737 +0000 UTC m=+2055.944194107" Oct 30 19:39:08.260858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.260812 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:08.261055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.260866 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:08.295025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.288709 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:08.323335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.323308 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.323476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.323461 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.323849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.323826 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:08.323981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.323909 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.323981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.323949 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324072 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324240 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324452 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324514 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324330 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.324647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324479 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.324647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324561 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\") pod \"d5cc01d8-2100-4da2-8908-b792cc280149\" (UID: \"d5cc01d8-2100-4da2-8908-b792cc280149\") " Oct 30 19:39:08.324765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.324747 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325002 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325372 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325398 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325416 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc01d8-2100-4da2-8908-b792cc280149-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325433 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.325488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.325448 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.326294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.326268 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "kube-api-access-6ctv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.326802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.326776 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:08.327166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.327136 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.327587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.327570 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy" pod "d5cc01d8-2100-4da2-8908-b792cc280149" (UID: "d5cc01d8-2100-4da2-8908-b792cc280149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.428178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.428081 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc01d8-2100-4da2-8908-b792cc280149-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.428178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.428124 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.428178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.428135 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc01d8-2100-4da2-8908-b792cc280149-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.428178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.428144 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctv2\" (UniqueName: \"kubernetes.io/projected/d5cc01d8-2100-4da2-8908-b792cc280149-kube-api-access-6ctv2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.611598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.611570 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:39:08.718644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.718604 4990 generic.go:356] "Generic (PLEG): container finished" podID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerID="ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147" exitCode=0 Oct 30 19:39:08.719136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.719075 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" event={"ID":"bbb7b181-fefc-4d27-8ee3-1188824ee897","Type":"ContainerDied","Data":"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147"} Oct 30 19:39:08.719339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.719324 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" event={"ID":"bbb7b181-fefc-4d27-8ee3-1188824ee897","Type":"ContainerDied","Data":"31499956928e2bd3ada76eb03bc2610b4bba9f5cf27499d3b44e7b0889e93967"} Oct 30 19:39:08.719454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.719443 4990 scope.go:122] "RemoveContainer" containerID="ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147" Oct 30 19:39:08.719707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.719694 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" Oct 30 19:39:08.724166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.723248 4990 generic.go:356] "Generic (PLEG): container finished" podID="d5cc01d8-2100-4da2-8908-b792cc280149" containerID="2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2" exitCode=0 Oct 30 19:39:08.724166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.723306 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" Oct 30 19:39:08.724166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.723338 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" event={"ID":"d5cc01d8-2100-4da2-8908-b792cc280149","Type":"ContainerDied","Data":"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2"} Oct 30 19:39:08.724166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.723492 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph" event={"ID":"d5cc01d8-2100-4da2-8908-b792cc280149","Type":"ContainerDied","Data":"7841bfd8cb8ab7758a8e42e2ac661eb43d954e09f49f27d082a301ee85ac8871"} Oct 30 19:39:08.732875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.732845 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.732913 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733037 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733130 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733257 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733309 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733423 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733491 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.733753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.733623 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\") pod \"bbb7b181-fefc-4d27-8ee3-1188824ee897\" (UID: \"bbb7b181-fefc-4d27-8ee3-1188824ee897\") " Oct 30 19:39:08.734279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.734045 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.734657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.734476 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.734657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.734601 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.735145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.734982 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:08.735221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.735198 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.735696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.735675 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.749135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.737650 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.758579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.758043 4990 scope.go:122] "RemoveContainer" containerID="ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147" Oct 30 19:39:08.759628 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:08.759468 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147\": container with ID starting with ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147 not found: ID does not exist" containerID="ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147" Oct 30 19:39:08.759628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.759511 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147"} err="failed to get container status \"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147\": rpc error: code = NotFound desc = could not find container \"ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147\": container with ID starting with ec2b74d127cced7bb9d336d0ebf4b6ed706cb859dbf961a44c390a1988cc2147 not found: ID does not exist" Oct 30 19:39:08.759628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.759536 4990 scope.go:122] "RemoveContainer" containerID="2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2" Oct 30 19:39:08.784830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.782799 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:08.792773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.792721 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.793185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.793085 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd" pod "bbb7b181-fefc-4d27-8ee3-1188824ee897" (UID: "bbb7b181-fefc-4d27-8ee3-1188824ee897"). InnerVolumeSpecName "kube-api-access-26qdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.817197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.817167 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:39:08.824073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.823942 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-bk8ph"] Oct 30 19:39:08.829024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.829002 4990 scope.go:122] "RemoveContainer" containerID="2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2" Oct 30 19:39:08.829631 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:08.829593 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2\": container with ID starting with 2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2 not found: ID does not exist" containerID="2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2" Oct 30 19:39:08.829813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.829786 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2"} err="failed to get container status \"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2\": rpc error: code = NotFound desc = could not find container \"2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2\": container with ID starting with 2f23e5a601c992c7ef67db0d6f021f9831e2d5a05809383737d4054326c646c2 not found: ID does not exist" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836666 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836696 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836710 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836722 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836736 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb7b181-fefc-4d27-8ee3-1188824ee897-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836749 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836762 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qdd\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-kube-api-access-26qdd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.836801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:08.836779 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb7b181-fefc-4d27-8ee3-1188824ee897-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.007480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.007452 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:39:09.107383 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:09.107299 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb7b181_fefc_4d27_8ee3_1188824ee897.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:39:09.107533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.107492 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:39:09.113405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.113378 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6"] Oct 30 19:39:09.140585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140556 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140609 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140639 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140679 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140774 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140855 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.140893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140860 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:09.141050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.140983 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.141173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141127 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.141233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141202 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\") pod \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\" (UID: \"1f6ceb2b-642a-499e-ae5a-46d851ba2b57\") " Oct 30 19:39:09.141291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141233 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:09.141437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141416 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:09.141630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141601 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:09.141788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.141767 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:09.142049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.142026 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.142181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.142052 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.142181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.142067 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.142181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.142082 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.142181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.142111 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.143288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.143264 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "kube-api-access-kwt4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:09.143965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.143939 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:09.144370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.144341 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:09.145250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.145223 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy" pod "1f6ceb2b-642a-499e-ae5a-46d851ba2b57" (UID: "1f6ceb2b-642a-499e-ae5a-46d851ba2b57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:09.243413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.243376 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.243413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.243412 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwt4s\" (UniqueName: \"kubernetes.io/projected/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-kube-api-access-kwt4s\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.243620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.243427 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.243620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.243441 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ceb2b-642a-499e-ae5a-46d851ba2b57-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.261056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.261019 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:09.261246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.261115 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:09.403495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.403401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:09.566562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.566528 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:09.580197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.580160 4990 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:39:09.580487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.580462 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-4m4r6" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:09.735321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.735233 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerID="5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea" exitCode=0 Oct 30 19:39:09.735321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.735282 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerDied","Data":"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea"} Oct 30 19:39:09.735321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.735305 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" Oct 30 19:39:09.735550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.735352 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8" event={"ID":"1f6ceb2b-642a-499e-ae5a-46d851ba2b57","Type":"ContainerDied","Data":"734edba85becf4151962d64870ea33fb3440e3b9a69038da029de8703460bbba"} Oct 30 19:39:09.735550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.735377 4990 scope.go:122] "RemoveContainer" containerID="5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea" Oct 30 19:39:09.748609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.748589 4990 scope.go:122] "RemoveContainer" containerID="c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7" Oct 30 19:39:09.759795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.759771 4990 scope.go:122] "RemoveContainer" containerID="5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea" Oct 30 19:39:09.760149 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:09.760122 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea\": container with ID starting with 5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea not found: ID does not exist" containerID="5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea" Oct 30 19:39:09.760236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.760152 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea"} err="failed to get container status \"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea\": rpc error: code = NotFound desc = could not find container \"5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea\": container with ID starting with 5a366b304f4af8e5457edf9b943086376d3c9fa6564e93c1fb612bde3a5bc1ea not found: ID does not exist" Oct 30 19:39:09.760236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.760170 4990 scope.go:122] "RemoveContainer" containerID="c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7" Oct 30 19:39:09.760440 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:09.760423 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7\": container with ID starting with c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7 not found: ID does not exist" containerID="c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7" Oct 30 19:39:09.760479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.760446 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7"} err="failed to get container status \"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7\": rpc error: code = NotFound desc = could not find container \"c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7\": container with ID starting with c7a63373a8e0842d569c745c567ac9f462bae8738a327573e660d971e84846e7 not found: ID does not exist" Oct 30 19:39:09.796177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.796140 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:39:09.800642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:09.800618 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ccf8bd61-openshift-default-7f595f9845-5bsp8"] Oct 30 19:39:10.225968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.225890 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:10.260514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.260464 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:10.260699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.260542 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:10.408212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.408170 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" path="/var/lib/kubelet/pods/1f6ceb2b-642a-499e-ae5a-46d851ba2b57/volumes" Oct 30 19:39:10.408888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.408864 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" path="/var/lib/kubelet/pods/bbb7b181-fefc-4d27-8ee3-1188824ee897/volumes" Oct 30 19:39:10.409466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:10.409441 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" path="/var/lib/kubelet/pods/d5cc01d8-2100-4da2-8908-b792cc280149/volumes" Oct 30 19:39:11.122304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.122248 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:11.225868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.225829 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:39:11.226875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226852 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" Oct 30 19:39:11.226935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226878 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" Oct 30 19:39:11.226935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226895 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.226935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226903 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.226935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226919 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" Oct 30 19:39:11.226935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226928 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" Oct 30 19:39:11.227155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226956 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.227155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.226963 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.227229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.227164 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cc01d8-2100-4da2-8908-b792cc280149" containerName="istio-proxy" Oct 30 19:39:11.227229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.227174 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.227229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.227185 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb7b181-fefc-4d27-8ee3-1188824ee897" containerName="istio-proxy" Oct 30 19:39:11.233360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.233326 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.255734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.255707 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-614c02\"" Oct 30 19:39:11.255982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.255840 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-614c02-dockercfg-kdnn5\"" Oct 30 19:39:11.260847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.260817 4990 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:39:11.260984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.260881 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 19:39:11.264525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.264496 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:39:11.284680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.284539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:11.395177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.394662 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.395177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.394888 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvxs\" (UniqueName: \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.500110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.496051 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvxs\" (UniqueName: \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.557085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.556701 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.558246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.558204 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.587115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.586235 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:39:11.600160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.587902 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6ceb2b-642a-499e-ae5a-46d851ba2b57" containerName="istio-proxy" Oct 30 19:39:11.601363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.601329 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvxs\" (UniqueName: \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\") pod \"payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.603495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.603471 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:39:11.603596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.603541 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.760948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.760849 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.760948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.760932 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wljm\" (UniqueName: \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.857758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.857713 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:11.872164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.872124 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wljm\" (UniqueName: \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.872460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.872439 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.873205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.873171 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.891688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.891659 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wljm\" (UniqueName: \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\") pod \"payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:11.933422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:11.933398 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:12.173982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.173952 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:39:12.175333 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:39:12.175306 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5d0de9_6995_4b3f_ac55_7b3c42a660a4.slice/crio-73ec5fa5b7ffe0b1b1bf5fd8ff45bc4986131da329cbc69e1af2b7122d5f86a0 WatchSource:0}: Error finding container 73ec5fa5b7ffe0b1b1bf5fd8ff45bc4986131da329cbc69e1af2b7122d5f86a0: Status 404 returned error can't find the container with id 73ec5fa5b7ffe0b1b1bf5fd8ff45bc4986131da329cbc69e1af2b7122d5f86a0 Oct 30 19:39:12.267615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.267585 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:12.268828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.268631 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:12.269588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.269569 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:39:12.269773 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:39:12.269749 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19088b0a_79fa_4237_af46_4ed8847e00df.slice/crio-1bc4abb3afb60d43e8afb4ce357da212a59b9dc4cf8359682075b10c41c061ed WatchSource:0}: Error finding container 1bc4abb3afb60d43e8afb4ce357da212a59b9dc4cf8359682075b10c41c061ed: Status 404 returned error can't find the container with id 1bc4abb3afb60d43e8afb4ce357da212a59b9dc4cf8359682075b10c41c061ed Oct 30 19:39:12.273349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.273327 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:39:12.330738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.330562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:12.769831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.769790 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" event={"ID":"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4","Type":"ContainerStarted","Data":"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce"} Oct 30 19:39:12.770344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.769858 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" event={"ID":"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4","Type":"ContainerStarted","Data":"73ec5fa5b7ffe0b1b1bf5fd8ff45bc4986131da329cbc69e1af2b7122d5f86a0"} Oct 30 19:39:12.770344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.769887 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:12.771568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.771540 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" event={"ID":"19088b0a-79fa-4237-af46-4ed8847e00df","Type":"ContainerStarted","Data":"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc"} Oct 30 19:39:12.771674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.771592 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" event={"ID":"19088b0a-79fa-4237-af46-4ed8847e00df","Type":"ContainerStarted","Data":"1bc4abb3afb60d43e8afb4ce357da212a59b9dc4cf8359682075b10c41c061ed"} Oct 30 19:39:12.771674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.771637 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:12.810612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.810536 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" podStartSLOduration=1.8105185769999999 podStartE2EDuration="1.810518577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:12.807352235 +0000 UTC m=+2060.518877605" watchObservedRunningTime="2026-10-30 19:39:12.810518577 +0000 UTC m=+2060.522043944" Oct 30 19:39:12.848366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:12.848319 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" podStartSLOduration=1.848306491 podStartE2EDuration="1.848306491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:12.844217599 +0000 UTC m=+2060.555742998" watchObservedRunningTime="2026-10-30 19:39:12.848306491 +0000 UTC m=+2060.559831856" Oct 30 19:39:14.271456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:14.271415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:14.743957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:14.743928 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:39:15.845675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:15.845640 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:17.612157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:17.609328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.219850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:19.219819 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.367947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:19.367897 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.003541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.003503 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.152853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.152815 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.350988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.350947 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:39:21.357332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.357308 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.376348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.376325 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a7b7bc-openshift-default-dockercfg-dgqfw\"" Oct 30 19:39:21.393247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.393217 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:39:21.458285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458251 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458311 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458536 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458594 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458694 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69k6\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458763 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458822 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458841 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.458975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.458955 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.459952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.459925 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.573793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.573751 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574041 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574258 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69k6\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574441 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574574 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574687 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.574883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.574867 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.575040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.575026 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.575204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.575185 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.589785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.589752 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.590340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.590315 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.590711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.590691 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.591632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.591607 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.595708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.595674 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.601661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.598887 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.617110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.615537 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69k6\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.618748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.618672 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.623191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.622287 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.679008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.678969 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:21.693596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:21.693562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:22.083017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.082981 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:39:22.084679 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:39:22.084635 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635573ca_0a7b_4a62_85d2_17ade59afb1f.slice/crio-9164125e612f3c63521546240a5015f827ec5366bdd1c56bdc069c5a6ae1006b WatchSource:0}: Error finding container 9164125e612f3c63521546240a5015f827ec5366bdd1c56bdc069c5a6ae1006b: Status 404 returned error can't find the container with id 9164125e612f3c63521546240a5015f827ec5366bdd1c56bdc069c5a6ae1006b Oct 30 19:39:22.087998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.087954 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:22.088086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.088069 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:22.088186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.088140 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:39:22.874927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.874892 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerStarted","Data":"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60"} Oct 30 19:39:22.875307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.875271 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerStarted","Data":"9164125e612f3c63521546240a5015f827ec5366bdd1c56bdc069c5a6ae1006b"} Oct 30 19:39:22.932554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:22.932503 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podStartSLOduration=1.932488041 podStartE2EDuration="1.932488041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:22.929022277 +0000 UTC m=+2070.640547644" watchObservedRunningTime="2026-10-30 19:39:22.932488041 +0000 UTC m=+2070.644013407" Oct 30 19:39:23.026338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.026300 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:23.164139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.163992 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:23.679974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.679936 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:23.687515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.687466 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:23.789824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.789791 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:39:23.790007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.789976 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:39:23.857116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:23.856698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:24.167031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:24.165452 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:24.174257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:24.174224 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:39:24.855314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:24.840702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:25.419315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:25.419276 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:25.582107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:25.573561 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:27.455488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.439237 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:27.557147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.557117 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:39:27.562533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.562508 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.603841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.603812 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a7b7bc\"" Oct 30 19:39:27.643253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.616772 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:27.657368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.657340 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:39:27.664566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.664531 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:39:27.814181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.762053 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.814181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.762126 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2ng\" (UniqueName: \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.814181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.763010 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:27.917423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.917382 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:27.917710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.917690 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xp6p\" (UniqueName: \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:27.918147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.918123 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.918292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.918274 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2ng\" (UniqueName: \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.919815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.919790 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:27.989118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:27.952807 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a7b7bc-dockercfg-d2cmj\"" Oct 30 19:39:28.020043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.019217 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:39:28.031884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.031769 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:28.036131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.035694 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:28.036377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.036122 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xp6p\" (UniqueName: \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:28.067158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.048162 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:39:28.067158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.048390 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" containerID="cri-o://955161f8ed4c9e0ebb8ddf1a9a2b1d0b13b5b07760efd16eae287c45f26ac8bc" gracePeriod=30 Oct 30 19:39:28.067158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.062502 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xp6p\" (UniqueName: \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\") pod \"payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:28.130506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.130414 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2ng\" (UniqueName: \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\") pod \"payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:28.131215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.131186 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:28.131970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.131888 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:28.134212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.134135 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:28.733132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.731502 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:28.779249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:28.779201 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:29.174009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.173972 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:39:29.240214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.240147 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:39:29.613186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.613145 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" event={"ID":"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8","Type":"ContainerStarted","Data":"400d1e28c4a453370d8ce65d469515ff45742230ebd08d9f2f76caa7ac78161e"} Oct 30 19:39:29.613391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.613207 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" event={"ID":"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8","Type":"ContainerStarted","Data":"574f711d21fca48f6182d27adc3ecd64ef9fd52238b8cabfcd7747d239ccd53f"} Oct 30 19:39:29.613391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.613237 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:29.614927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.614899 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" event={"ID":"e5e339a2-eb62-4b7a-a59f-3ddee83607eb","Type":"ContainerStarted","Data":"e7d497b187585227b3167f035848ac441d238622ee4f8974f4a8c13e21ee2af5"} Oct 30 19:39:29.615060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.614945 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" event={"ID":"e5e339a2-eb62-4b7a-a59f-3ddee83607eb","Type":"ContainerStarted","Data":"ff57b24f6c7faa11380254d1b7541c8a81fa090e71ab208b9e91df5cc948f712"} Oct 30 19:39:29.615060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.615027 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:29.653269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.653223 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" podStartSLOduration=2.653210337 podStartE2EDuration="2.653210337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:29.64936681 +0000 UTC m=+2077.360892178" watchObservedRunningTime="2026-10-30 19:39:29.653210337 +0000 UTC m=+2077.364735702" Oct 30 19:39:29.688876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.688820 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" podStartSLOduration=2.688802172 podStartE2EDuration="2.688802172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:29.684520737 +0000 UTC m=+2077.396046095" watchObservedRunningTime="2026-10-30 19:39:29.688802172 +0000 UTC m=+2077.400327541" Oct 30 19:39:29.988418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:29.988332 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:30.555745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:30.555709 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:30.706908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:30.706869 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:31.926543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:31.926503 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:32.490891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:32.490849 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:39:32.491134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:32.490913 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:39:33.649780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.649747 4990 generic.go:356] "Generic (PLEG): container finished" podID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerID="955161f8ed4c9e0ebb8ddf1a9a2b1d0b13b5b07760efd16eae287c45f26ac8bc" exitCode=0 Oct 30 19:39:33.650187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.649822 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" event={"ID":"53817bca-3067-4bbc-9d96-f74a87a56c1a","Type":"ContainerDied","Data":"955161f8ed4c9e0ebb8ddf1a9a2b1d0b13b5b07760efd16eae287c45f26ac8bc"} Oct 30 19:39:33.836990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.836449 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:33.874625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.874580 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:39:33.962844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.962744 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.962844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.962801 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.962880 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.962941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963219 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:33.963314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963242 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:33.963472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963452 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963513 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963656 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963713 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963846 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\") pod \"53817bca-3067-4bbc-9d96-f74a87a56c1a\" (UID: \"53817bca-3067-4bbc-9d96-f74a87a56c1a\") " Oct 30 19:39:33.963984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.963949 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:33.964247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964221 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:33.964349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964258 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:33.964760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964735 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:33.964760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964763 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:33.964951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964776 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53817bca-3067-4bbc-9d96-f74a87a56c1a-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:33.964951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964786 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:33.964951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.964795 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:33.966488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.966459 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:33.968659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.968500 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:33.968761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.968654 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "kube-api-access-z92l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:33.968761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:33.968731 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy" pod "53817bca-3067-4bbc-9d96-f74a87a56c1a" (UID: "53817bca-3067-4bbc-9d96-f74a87a56c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:34.066679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.066627 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:34.066679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.066674 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:34.066679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.066683 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53817bca-3067-4bbc-9d96-f74a87a56c1a-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:34.066679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.066694 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z92l2\" (UniqueName: \"kubernetes.io/projected/53817bca-3067-4bbc-9d96-f74a87a56c1a-kube-api-access-z92l2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:34.659568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.659530 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" event={"ID":"53817bca-3067-4bbc-9d96-f74a87a56c1a","Type":"ContainerDied","Data":"c219ca27b1be79c4edd64ae564c74e9e42c340ee617c7558e10b0a424d5899cb"} Oct 30 19:39:34.659568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.659555 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k" Oct 30 19:39:34.660086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.659601 4990 scope.go:122] "RemoveContainer" containerID="955161f8ed4c9e0ebb8ddf1a9a2b1d0b13b5b07760efd16eae287c45f26ac8bc" Oct 30 19:39:34.720006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.719926 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:39:34.727869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:34.727831 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-9bt2k"] Oct 30 19:39:35.098762 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:35.098722 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:39:35.467172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:35.455773 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:36.408137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:36.408086 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" path="/var/lib/kubelet/pods/53817bca-3067-4bbc-9d96-f74a87a56c1a/volumes" Oct 30 19:39:37.056002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:37.055958 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.135187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:38.135147 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.273565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:38.273520 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:38.950635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:38.949591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:40.625682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:40.625653 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:39:40.626116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:40.625727 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:39:40.644056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:40.644021 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:42.392646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:42.392610 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:42.983981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:42.983939 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:39:42.984245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:42.984211 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" containerID="cri-o://6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840" gracePeriod=30 Oct 30 19:39:44.096673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:44.096634 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.703035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:45.702997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:47.442672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:47.442628 4990 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:39:47.443071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:47.442700 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:39:47.560181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:47.560141 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:47.759643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:47.759544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:48.290490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.290464 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:39:48.408698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.408548 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.408698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.408627 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.409060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.408984 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409254 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409372 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409462 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409503 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409656 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.410596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.409731 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\") pod \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\" (UID: \"59350c6c-5729-4e1e-9e47-ff76ddd65c12\") " Oct 30 19:39:48.411584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.411193 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:48.412210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.411997 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:48.413897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.413867 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:48.414659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.414633 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:48.414772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.414743 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:48.422634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.422592 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "kube-api-access-dgdpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:48.423651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.423593 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:48.423948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.423919 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:48.424218 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.424076 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token" pod "59350c6c-5729-4e1e-9e47-ff76ddd65c12" (UID: "59350c6c-5729-4e1e-9e47-ff76ddd65c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:48.512257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512222 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgdpt\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-kube-api-access-dgdpt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512256 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512281 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512298 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512313 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512327 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512340 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512353 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59350c6c-5729-4e1e-9e47-ff76ddd65c12-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.512666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.512366 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59350c6c-5729-4e1e-9e47-ff76ddd65c12-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:39:48.848114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.847321 4990 generic.go:356] "Generic (PLEG): container finished" podID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerID="6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840" exitCode=0 Oct 30 19:39:48.848114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.847525 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" event={"ID":"59350c6c-5729-4e1e-9e47-ff76ddd65c12","Type":"ContainerDied","Data":"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840"} Oct 30 19:39:48.848114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.847609 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" event={"ID":"59350c6c-5729-4e1e-9e47-ff76ddd65c12","Type":"ContainerDied","Data":"386156c756df8a42690e4d47ade9937880ac13c09935e09854c8161243581ee3"} Oct 30 19:39:48.848114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.847647 4990 scope.go:122] "RemoveContainer" containerID="6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840" Oct 30 19:39:48.848114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.847929 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz" Oct 30 19:39:48.884166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.884071 4990 scope.go:122] "RemoveContainer" containerID="6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840" Oct 30 19:39:48.884538 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:39:48.884499 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840\": container with ID starting with 6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840 not found: ID does not exist" containerID="6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840" Oct 30 19:39:48.884693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.884549 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840"} err="failed to get container status \"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840\": rpc error: code = NotFound desc = could not find container \"6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840\": container with ID starting with 6e5950f2e519b310f59c97b28316770b16e7ce4fd6caf3922ce50c769c9b3840 not found: ID does not exist" Oct 30 19:39:48.920162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.917584 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:39:48.923916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:48.923883 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ldmzz"] Oct 30 19:39:49.722882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:49.719653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:50.404211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:50.404173 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" path="/var/lib/kubelet/pods/59350c6c-5729-4e1e-9e47-ff76ddd65c12/volumes" Oct 30 19:39:51.095819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:51.095755 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:51.595216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:51.590961 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:51.752460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:51.752425 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:52.099453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:52.099406 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:52.334406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:52.334367 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:53.577446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:53.577408 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:54.211243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:54.211210 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:39:54.220965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:54.220941 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:39:54.224683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:54.224659 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:39:54.235685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:54.235662 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:39:55.454190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:55.448546 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:57.017317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:57.014810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:58.617316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:39:58.617222 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.169504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:00.169459 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:00.301211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:00.301172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:01.982568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:01.982524 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:03.870425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:03.870386 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:04.744458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:04.744397 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.101606 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:05.101570 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:40:05.193125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:05.193055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.443817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:05.443736 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.657038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:05.656997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:05.833743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:05.833704 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.006430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:06.006393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.180485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:06.180396 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.400309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:06.400268 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:06.587143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:06.585496 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:07.764348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.764305 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:07.906936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.906902 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:40:07.907701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907685 4990 cpu_manager.go:414] "containerMap: removing container" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" Oct 30 19:40:07.907778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907702 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" Oct 30 19:40:07.907778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907728 4990 cpu_manager.go:414] "containerMap: removing container" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" Oct 30 19:40:07.907778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907733 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" Oct 30 19:40:07.907893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907838 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="53817bca-3067-4bbc-9d96-f74a87a56c1a" containerName="istio-proxy" Oct 30 19:40:07.907893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.907854 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="59350c6c-5729-4e1e-9e47-ff76ddd65c12" containerName="istio-proxy" Oct 30 19:40:07.914954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.914922 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:07.934380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.934352 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:40:07.938916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.938892 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-469724-dockercfg-m8zj6\"" Oct 30 19:40:07.939047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:07.938979 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-469724\"" Oct 30 19:40:08.125179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.092758 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:08.131620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.131582 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncch\" (UniqueName: \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.132027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.132002 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.136330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.136296 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:40:08.201103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.201051 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:40:08.201571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.201496 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.233803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.233771 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.234107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.234071 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncch\" (UniqueName: \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.235436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.235410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.251895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.251860 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncch\" (UniqueName: \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\") pod \"payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.336049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.336002 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.336262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.336194 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vm2b\" (UniqueName: \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.435384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.435299 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:08.437800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.437761 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.437951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.437867 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vm2b\" (UniqueName: \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.438455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.438437 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.508115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.507801 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vm2b\" (UniqueName: \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\") pod \"payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.530547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.529393 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:08.567181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.561217 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:08.979675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.979640 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:40:08.983712 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:08.983682 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode664fa9a_5236_445a_9d95_5fa23503dbb6.slice/crio-fd073a287c688ba0fbe41c0010fc3b7bb6b50096c4ed5e19f4bb24ccb0fa331c WatchSource:0}: Error finding container fd073a287c688ba0fbe41c0010fc3b7bb6b50096c4ed5e19f4bb24ccb0fa331c: Status 404 returned error can't find the container with id fd073a287c688ba0fbe41c0010fc3b7bb6b50096c4ed5e19f4bb24ccb0fa331c Oct 30 19:40:08.996269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:08.996239 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:40:09.002314 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:09.002286 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb083da_c973_4f04_baff_a9e4eb0a6c1c.slice/crio-359449f7db8fb1409cf9126231661254198ad0db1795d25ebd6f8318daf257b6 WatchSource:0}: Error finding container 359449f7db8fb1409cf9126231661254198ad0db1795d25ebd6f8318daf257b6: Status 404 returned error can't find the container with id 359449f7db8fb1409cf9126231661254198ad0db1795d25ebd6f8318daf257b6 Oct 30 19:40:09.208017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:09.204455 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" event={"ID":"e664fa9a-5236-445a-9d95-5fa23503dbb6","Type":"ContainerStarted","Data":"fd073a287c688ba0fbe41c0010fc3b7bb6b50096c4ed5e19f4bb24ccb0fa331c"} Oct 30 19:40:09.208017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:09.206417 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" event={"ID":"dcb083da-c973-4f04-baff-a9e4eb0a6c1c","Type":"ContainerStarted","Data":"359449f7db8fb1409cf9126231661254198ad0db1795d25ebd6f8318daf257b6"} Oct 30 19:40:09.493377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:09.490582 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:10.219516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.219477 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" event={"ID":"dcb083da-c973-4f04-baff-a9e4eb0a6c1c","Type":"ContainerStarted","Data":"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c"} Oct 30 19:40:10.219992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.219567 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:10.221433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.221405 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" event={"ID":"e664fa9a-5236-445a-9d95-5fa23503dbb6","Type":"ContainerStarted","Data":"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf"} Oct 30 19:40:10.221576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.221513 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:10.253859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.253805 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" podStartSLOduration=3.253790083 podStartE2EDuration="3.253790083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:10.253059963 +0000 UTC m=+2117.964585329" watchObservedRunningTime="2026-10-30 19:40:10.253790083 +0000 UTC m=+2117.965315449" Oct 30 19:40:10.289128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.289045 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" podStartSLOduration=3.289030393 podStartE2EDuration="3.289030393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:10.286713009 +0000 UTC m=+2117.998238375" watchObservedRunningTime="2026-10-30 19:40:10.289030393 +0000 UTC m=+2118.000555759" Oct 30 19:40:10.577111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.571854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:10.729062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:10.729022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:12.537115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:12.536076 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:12.872124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:12.871183 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:40:12.872124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:12.871250 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.936457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:13.936405 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.090699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.090660 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:40:14.093321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.093288 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.096750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.096729 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.114438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.114410 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:40:14.118761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.118124 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-09feeb-dockercfg-7q47n\"" Oct 30 19:40:14.118761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.118187 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-09feeb\"" Oct 30 19:40:14.219236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.213487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.219740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.219702 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46l2\" (UniqueName: \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.351242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.351202 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.351459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.351261 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46l2\" (UniqueName: \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.352208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.352178 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.369367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.369333 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46l2\" (UniqueName: \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\") pod \"payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.413657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.413623 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:14.748546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:14.748517 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:40:14.749298 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:14.749273 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afe5982_06f8_4993_99f9_8e268b1397f0.slice/crio-057d2507a0fe2986070ac26cdb347cffd4dbf65d42ccca9eca83689c36c9574e WatchSource:0}: Error finding container 057d2507a0fe2986070ac26cdb347cffd4dbf65d42ccca9eca83689c36c9574e: Status 404 returned error can't find the container with id 057d2507a0fe2986070ac26cdb347cffd4dbf65d42ccca9eca83689c36c9574e Oct 30 19:40:15.314155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:15.314112 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" event={"ID":"3afe5982-06f8-4993-99f9-8e268b1397f0","Type":"ContainerStarted","Data":"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912"} Oct 30 19:40:15.314563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:15.314218 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:15.314563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:15.314246 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" event={"ID":"3afe5982-06f8-4993-99f9-8e268b1397f0","Type":"ContainerStarted","Data":"057d2507a0fe2986070ac26cdb347cffd4dbf65d42ccca9eca83689c36c9574e"} Oct 30 19:40:15.347940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:15.347892 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" podStartSLOduration=1.347880581 podStartE2EDuration="1.347880581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:15.346765064 +0000 UTC m=+2123.058290435" watchObservedRunningTime="2026-10-30 19:40:15.347880581 +0000 UTC m=+2123.059405925" Oct 30 19:40:15.420897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:15.420856 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:17.070427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:17.070390 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:17.346177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:17.343821 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.480755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:18.480714 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:40:18.481263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:18.480967 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" containerID="cri-o://84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752" gracePeriod=30 Oct 30 19:40:19.202307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:19.202258 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.899767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:20.899731 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:21.254394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:21.254300 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:40:21.254553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:21.254457 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:40:22.648053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:22.648012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.098676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.098628 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.261381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.261348 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.807807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.807777 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858144 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858362 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858442 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858502 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858550 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858587 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858615 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858653 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.859530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.858682 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\") pod \"906cda27-24c5-413b-8166-ea2439390411\" (UID: \"906cda27-24c5-413b-8166-ea2439390411\") " Oct 30 19:40:23.860402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.860195 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.860649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.860519 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.864656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.864387 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.872376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.872345 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:23.873816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.873383 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:23.882861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.882838 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:23.883197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.883175 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "kube-api-access-9vvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:23.884501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.884478 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.886221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.886190 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs" pod "906cda27-24c5-413b-8166-ea2439390411" (UID: "906cda27-24c5-413b-8166-ea2439390411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.960257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960224 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/906cda27-24c5-413b-8166-ea2439390411-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960254 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960265 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/906cda27-24c5-413b-8166-ea2439390411-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960278 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvxk\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-kube-api-access-9vvxk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960292 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960303 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/906cda27-24c5-413b-8166-ea2439390411-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960314 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960324 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.960493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:23.960336 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/906cda27-24c5-413b-8166-ea2439390411-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.419573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.419536 4990 generic.go:356] "Generic (PLEG): container finished" podID="906cda27-24c5-413b-8166-ea2439390411" containerID="84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752" exitCode=0 Oct 30 19:40:24.419697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.419595 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" event={"ID":"906cda27-24c5-413b-8166-ea2439390411","Type":"ContainerDied","Data":"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752"} Oct 30 19:40:24.419697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.419648 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" event={"ID":"906cda27-24c5-413b-8166-ea2439390411","Type":"ContainerDied","Data":"d40cee6d2c85fa77aca48a1b90043b7dab7da22c400ba8033e7da00c82f90929"} Oct 30 19:40:24.419697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.419650 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5" Oct 30 19:40:24.419697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.419670 4990 scope.go:122] "RemoveContainer" containerID="84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752" Oct 30 19:40:24.431221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.431200 4990 scope.go:122] "RemoveContainer" containerID="84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752" Oct 30 19:40:24.431519 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:24.431496 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752\": container with ID starting with 84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752 not found: ID does not exist" containerID="84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752" Oct 30 19:40:24.431634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.431523 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752"} err="failed to get container status \"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752\": rpc error: code = NotFound desc = could not find container \"84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752\": container with ID starting with 84abcab7eb60bbf00ad6439a7a2c47d963b280e8d158407d79f18481d7796752 not found: ID does not exist" Oct 30 19:40:24.481559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.481524 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:40:24.491196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.491166 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-5b5c5"] Oct 30 19:40:24.512111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:24.499387 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.196747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:26.196708 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.326424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:26.326394 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:40:26.409847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:26.409808 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906cda27-24c5-413b-8166-ea2439390411" path="/var/lib/kubelet/pods/906cda27-24c5-413b-8166-ea2439390411/volumes" Oct 30 19:40:28.018668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:28.018608 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.906725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:29.906688 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.557471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:31.557428 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:33.096557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:33.096521 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:33.451161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:33.451059 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:35.031844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:35.031777 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:35.098855 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:35.098817 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:40:36.928904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:36.928869 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:38.903942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:38.903799 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:39.787942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:39.787903 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:40:39.859248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:39.859216 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:40.873139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:40.868965 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.617279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:42.617239 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.523558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:44.523466 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.831872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:44.831833 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:40:44.863195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:44.863162 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:40:45.013520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:45.013483 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.157453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:45.157365 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.282264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:46.282218 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.436468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:46.436429 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.737922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:46.737881 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.295170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:47.295127 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.476212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:47.476176 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:48.240614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:48.239757 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:49.832259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:49.832214 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" podUID="4c60a666-d1ed-40d1-86a9-6b07247e0617" containerName="payload-processing" containerID="cri-o://caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e" gracePeriod=25 Oct 30 19:40:49.863843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:49.863808 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" podUID="6b880856-0902-40a8-86a1-275db8694ad5" containerName="payload-pre-processing" containerID="cri-o://53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956" gracePeriod=25 Oct 30 19:40:50.189922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.189894 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:40:50.212270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.212240 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\" (UniqueName: \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\") pod \"4c60a666-d1ed-40d1-86a9-6b07247e0617\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " Oct 30 19:40:50.212437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.212420 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\") pod \"4c60a666-d1ed-40d1-86a9-6b07247e0617\" (UID: \"4c60a666-d1ed-40d1-86a9-6b07247e0617\") " Oct 30 19:40:50.213066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.212801 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume" pod "4c60a666-d1ed-40d1-86a9-6b07247e0617" (UID: "4c60a666-d1ed-40d1-86a9-6b07247e0617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:50.213066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.213051 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c60a666-d1ed-40d1-86a9-6b07247e0617-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:50.214669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.214646 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9" pod "4c60a666-d1ed-40d1-86a9-6b07247e0617" (UID: "4c60a666-d1ed-40d1-86a9-6b07247e0617"). InnerVolumeSpecName "kube-api-access-nt7k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:50.226133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.226081 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:40:50.314057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.313987 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\") pod \"6b880856-0902-40a8-86a1-275db8694ad5\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " Oct 30 19:40:50.314278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.314114 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\" (UniqueName: \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\") pod \"6b880856-0902-40a8-86a1-275db8694ad5\" (UID: \"6b880856-0902-40a8-86a1-275db8694ad5\") " Oct 30 19:40:50.314792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.314766 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume" pod "6b880856-0902-40a8-86a1-275db8694ad5" (UID: "6b880856-0902-40a8-86a1-275db8694ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:50.314968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.314850 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7k9\" (UniqueName: \"kubernetes.io/projected/4c60a666-d1ed-40d1-86a9-6b07247e0617-kube-api-access-nt7k9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:50.317229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.317201 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz" pod "6b880856-0902-40a8-86a1-275db8694ad5" (UID: "6b880856-0902-40a8-86a1-275db8694ad5"). InnerVolumeSpecName "kube-api-access-pbqwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:50.414173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.406799 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.418137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.417280 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b880856-0902-40a8-86a1-275db8694ad5-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:50.418137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.417316 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqwz\" (UniqueName: \"kubernetes.io/projected/6b880856-0902-40a8-86a1-275db8694ad5-kube-api-access-pbqwz\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:50.694344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.693853 4990 generic.go:356] "Generic (PLEG): container finished" podID="6b880856-0902-40a8-86a1-275db8694ad5" containerID="53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956" exitCode=0 Oct 30 19:40:50.694344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.693926 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" Oct 30 19:40:50.694344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.693963 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" event={"ID":"6b880856-0902-40a8-86a1-275db8694ad5","Type":"ContainerDied","Data":"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956"} Oct 30 19:40:50.694344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.694299 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr" event={"ID":"6b880856-0902-40a8-86a1-275db8694ad5","Type":"ContainerDied","Data":"022f98f81ff116bc6b39b652fc78053f3c6236d086816f4fb8db656d31760af1"} Oct 30 19:40:50.694681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.694350 4990 scope.go:122] "RemoveContainer" containerID="53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956" Oct 30 19:40:50.696972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.696947 4990 generic.go:356] "Generic (PLEG): container finished" podID="4c60a666-d1ed-40d1-86a9-6b07247e0617" containerID="caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e" exitCode=0 Oct 30 19:40:50.697131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.696983 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" event={"ID":"4c60a666-d1ed-40d1-86a9-6b07247e0617","Type":"ContainerDied","Data":"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e"} Oct 30 19:40:50.697131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.697022 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" event={"ID":"4c60a666-d1ed-40d1-86a9-6b07247e0617","Type":"ContainerDied","Data":"de48fefe71a05b714d9a5dedf31b1433e63fd6b992e88aee32b437bd79e19272"} Oct 30 19:40:50.697283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.697148 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2" Oct 30 19:40:50.710779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.710756 4990 scope.go:122] "RemoveContainer" containerID="53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956" Oct 30 19:40:50.711171 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:50.711144 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956\": container with ID starting with 53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956 not found: ID does not exist" containerID="53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956" Oct 30 19:40:50.711265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.711181 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956"} err="failed to get container status \"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956\": rpc error: code = NotFound desc = could not find container \"53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956\": container with ID starting with 53b4466657c6895b108762ab151333b0761a926a76df551534ec6b77698da956 not found: ID does not exist" Oct 30 19:40:50.711265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.711204 4990 scope.go:122] "RemoveContainer" containerID="caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e" Oct 30 19:40:50.722644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.722621 4990 scope.go:122] "RemoveContainer" containerID="caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e" Oct 30 19:40:50.722945 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:50.722924 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e\": container with ID starting with caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e not found: ID does not exist" containerID="caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e" Oct 30 19:40:50.723040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.722955 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e"} err="failed to get container status \"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e\": rpc error: code = NotFound desc = could not find container \"caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e\": container with ID starting with caa06eef70cddcc1d95b75892993b0c5f5e72ec04ce2653126ffa8e990d6db5e not found: ID does not exist" Oct 30 19:40:50.739851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.739821 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:40:50.745479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.745456 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a7c3b0-6f69f77fb-qjls2"] Oct 30 19:40:50.776409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.776371 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:40:50.785922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.785892 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a7c3b0-798c8b49bc-ghrtr"] Oct 30 19:40:50.943910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:50.943858 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.080700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:51.080660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.533169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:51.530667 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.690112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:51.689836 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.404661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:52.404630 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c60a666-d1ed-40d1-86a9-6b07247e0617" path="/var/lib/kubelet/pods/4c60a666-d1ed-40d1-86a9-6b07247e0617/volumes" Oct 30 19:40:52.405116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:52.405075 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b880856-0902-40a8-86a1-275db8694ad5" path="/var/lib/kubelet/pods/6b880856-0902-40a8-86a1-275db8694ad5/volumes" Oct 30 19:40:52.641118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:52.637132 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.807428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:52.807364 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.722108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:53.722061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:54.148778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.148728 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:54.221995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.221957 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:40:54.222743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222727 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6b880856-0902-40a8-86a1-275db8694ad5" containerName="payload-pre-processing" Oct 30 19:40:54.222795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222744 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b880856-0902-40a8-86a1-275db8694ad5" containerName="payload-pre-processing" Oct 30 19:40:54.222795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222762 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4c60a666-d1ed-40d1-86a9-6b07247e0617" containerName="payload-processing" Oct 30 19:40:54.222795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222767 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c60a666-d1ed-40d1-86a9-6b07247e0617" containerName="payload-processing" Oct 30 19:40:54.222906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222807 4990 cpu_manager.go:414] "containerMap: removing container" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" Oct 30 19:40:54.222906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222812 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" Oct 30 19:40:54.222969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222915 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="906cda27-24c5-413b-8166-ea2439390411" containerName="istio-proxy" Oct 30 19:40:54.222969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222928 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b880856-0902-40a8-86a1-275db8694ad5" containerName="payload-pre-processing" Oct 30 19:40:54.222969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.222938 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c60a666-d1ed-40d1-86a9-6b07247e0617" containerName="payload-processing" Oct 30 19:40:54.228199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.228175 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.243774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.243748 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:40:54.247688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.247663 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e3504b-dockercfg-nrxvq\"" Oct 30 19:40:54.247688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.247661 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e3504b\"" Oct 30 19:40:54.281289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.281253 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:40:54.293818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.293505 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlxv\" (UniqueName: \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.293818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.293634 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.307772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.301692 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.499324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.478490 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.499324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.479347 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.530172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.530127 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlxv\" (UniqueName: \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.583639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.583599 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlxv\" (UniqueName: \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\") pod \"payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.638408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.638378 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbt26\" (UniqueName: \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.638588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.638543 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.641494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.641466 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:40:54.739952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.739911 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbt26\" (UniqueName: \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.740504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.740270 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.740944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.740922 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.757481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.757406 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbt26\" (UniqueName: \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\") pod \"payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:54.811819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.811782 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" podUID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" containerName="main" containerID="cri-o://d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783" gracePeriod=45 Oct 30 19:40:54.846969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.845342 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:54.928058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:54.927817 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:55.153927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.153893 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:40:55.184846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.181783 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:40:55.184846 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:55.182884 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9254072_5a4b_4007_bc98_8e0b42946644.slice/crio-7390f8af094b45a5e231795b29eaa2b70ae9be5c3cdd6ebfcb7f500e8db0fc35 WatchSource:0}: Error finding container 7390f8af094b45a5e231795b29eaa2b70ae9be5c3cdd6ebfcb7f500e8db0fc35: Status 404 returned error can't find the container with id 7390f8af094b45a5e231795b29eaa2b70ae9be5c3cdd6ebfcb7f500e8db0fc35 Oct 30 19:40:55.256929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.256900 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257012 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257160 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.257307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257288 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257359 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257392 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257438 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\" (UniqueName: \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\") pod \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\" (UID: \"3d57598a-9ec8-48e1-8b02-9a956dcb0341\") " Oct 30 19:40:55.257611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257281 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.257809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.257777 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.258180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.258120 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.258180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.258146 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.258180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.258159 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.259461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.259434 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.259833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.259810 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "kube-api-access-dp8mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:55.260036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.260017 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs" pod "3d57598a-9ec8-48e1-8b02-9a956dcb0341" (UID: "3d57598a-9ec8-48e1-8b02-9a956dcb0341"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:55.288456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.288381 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:40:55.293973 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:55.293949 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d45b28d_aea3_4df1_98b9_64de5947a3ad.slice/crio-19d4e4eb3198f4ab5de076cbfa19e63c611d30a106e93e0fd3658a8d9c11eb93 WatchSource:0}: Error finding container 19d4e4eb3198f4ab5de076cbfa19e63c611d30a106e93e0fd3658a8d9c11eb93: Status 404 returned error can't find the container with id 19d4e4eb3198f4ab5de076cbfa19e63c611d30a106e93e0fd3658a8d9c11eb93 Oct 30 19:40:55.305229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.305204 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:55.364807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.359725 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d57598a-9ec8-48e1-8b02-9a956dcb0341-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.364807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.359762 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d57598a-9ec8-48e1-8b02-9a956dcb0341-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.364807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.359776 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp8mb\" (UniqueName: \"kubernetes.io/projected/3d57598a-9ec8-48e1-8b02-9a956dcb0341-kube-api-access-dp8mb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:40:55.451260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.451224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:55.800788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.800735 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" event={"ID":"2d45b28d-aea3-4df1-98b9-64de5947a3ad","Type":"ContainerStarted","Data":"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47"} Oct 30 19:40:55.801310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.800825 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" event={"ID":"2d45b28d-aea3-4df1-98b9-64de5947a3ad","Type":"ContainerStarted","Data":"19d4e4eb3198f4ab5de076cbfa19e63c611d30a106e93e0fd3658a8d9c11eb93"} Oct 30 19:40:55.801310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.800864 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:40:55.802961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.802933 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" event={"ID":"b9254072-5a4b-4007-bc98-8e0b42946644","Type":"ContainerStarted","Data":"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722"} Oct 30 19:40:55.803103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.802985 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" event={"ID":"b9254072-5a4b-4007-bc98-8e0b42946644","Type":"ContainerStarted","Data":"7390f8af094b45a5e231795b29eaa2b70ae9be5c3cdd6ebfcb7f500e8db0fc35"} Oct 30 19:40:55.803103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.803029 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:40:55.804581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.804558 4990 generic.go:356] "Generic (PLEG): container finished" podID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" containerID="d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783" exitCode=0 Oct 30 19:40:55.804695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.804600 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" Oct 30 19:40:55.804695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.804632 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" event={"ID":"3d57598a-9ec8-48e1-8b02-9a956dcb0341","Type":"ContainerDied","Data":"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783"} Oct 30 19:40:55.804695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.804672 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9" event={"ID":"3d57598a-9ec8-48e1-8b02-9a956dcb0341","Type":"ContainerDied","Data":"afe4e7df566816fd292c3dccc2239ea09620bae963e87cf74becc2664d7cce21"} Oct 30 19:40:55.804833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.804690 4990 scope.go:122] "RemoveContainer" containerID="d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783" Oct 30 19:40:55.838696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.838671 4990 scope.go:122] "RemoveContainer" containerID="d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783" Oct 30 19:40:55.839421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.839383 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" podStartSLOduration=1.839370033 podStartE2EDuration="1.839370033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:55.834764754 +0000 UTC m=+2163.546290126" watchObservedRunningTime="2026-10-30 19:40:55.839370033 +0000 UTC m=+2163.550895418" Oct 30 19:40:55.840125 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:40:55.840055 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783\": container with ID starting with d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783 not found: ID does not exist" containerID="d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783" Oct 30 19:40:55.840225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.840123 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783"} err="failed to get container status \"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783\": rpc error: code = NotFound desc = could not find container \"d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783\": container with ID starting with d39526217702c81bd88ee0cd5f56c619414220f085a7a7a595ffe05ea07d1783 not found: ID does not exist" Oct 30 19:40:55.873273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.873186 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" podStartSLOduration=1.8731732079999999 podStartE2EDuration="1.873173208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:55.870462981 +0000 UTC m=+2163.581988344" watchObservedRunningTime="2026-10-30 19:40:55.873173208 +0000 UTC m=+2163.584698575" Oct 30 19:40:55.904797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.904769 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:40:55.909123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:55.909083 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8e6b67/test-model-8e6b67-kserve-564bf858f9-4jpp9"] Oct 30 19:40:56.414149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:56.414111 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" path="/var/lib/kubelet/pods/3d57598a-9ec8-48e1-8b02-9a956dcb0341/volumes" Oct 30 19:40:57.404990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.404953 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:40:57.405759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.405743 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" containerName="main" Oct 30 19:40:57.405806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.405760 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" containerName="main" Oct 30 19:40:57.405883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.405873 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d57598a-9ec8-48e1-8b02-9a956dcb0341" containerName="main" Oct 30 19:40:57.410855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.410834 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.423255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.423231 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-09feeb-serving-cert\"" Oct 30 19:40:57.438972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.438944 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:40:57.516194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.516146 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4fb\" (UniqueName: \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.516387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.516323 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.516462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.516443 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.516679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.516661 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.604729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.604692 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.618495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.618458 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.618707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.618506 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4fb\" (UniqueName: \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.618707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.618614 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.618849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.618715 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.619331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.619305 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.621299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.621263 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.621395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.621265 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.636340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.636317 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4fb\" (UniqueName: \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\") pod \"maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.726812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.726714 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:57.820940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:57.820894 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.104444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.104410 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:40:58.106434 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:40:58.106407 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c9d084_1785_49c7_8cd9_8dd74e40c639.slice/crio-a9bd7ab7f23cea22d76f755347b256cdfa51b85f2cca9fcfcf1ec3fd0a011b1e WatchSource:0}: Error finding container a9bd7ab7f23cea22d76f755347b256cdfa51b85f2cca9fcfcf1ec3fd0a011b1e: Status 404 returned error can't find the container with id a9bd7ab7f23cea22d76f755347b256cdfa51b85f2cca9fcfcf1ec3fd0a011b1e Oct 30 19:40:58.107948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.107930 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:40:58.924108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.924056 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" event={"ID":"63c9d084-1785-49c7-8cd9-8dd74e40c639","Type":"ContainerStarted","Data":"03435b30a074153049136c59868c924e62f6517b284479041654759403822821"} Oct 30 19:40:58.924692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.924667 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" event={"ID":"63c9d084-1785-49c7-8cd9-8dd74e40c639","Type":"ContainerStarted","Data":"a9bd7ab7f23cea22d76f755347b256cdfa51b85f2cca9fcfcf1ec3fd0a011b1e"} Oct 30 19:40:58.926397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.926375 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:40:58.962519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:58.962451 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" podStartSLOduration=1.6176174190000001 podStartE2EDuration="1.962431503s" totalImagesPullingTime="344.814084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:58.960679151 +0000 UTC m=+2166.672204517" watchObservedRunningTime="2026-10-30 19:40:58.962431503 +0000 UTC m=+2166.673956870" Oct 30 19:40:59.200596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:59.200478 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.933742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:40:59.933704 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.096665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:00.096624 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.410980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:00.410945 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.546013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:00.545977 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.697765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:00.697668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.020339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:02.020292 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.164121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:02.162885 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.500792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:02.500324 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.001838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:03.001798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.175163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:03.174918 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.101039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:04.097424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.099282 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:05.099244 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:41:05.838342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:05.838306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.949520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:05.949490 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:41:06.818604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:06.818575 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:41:06.819016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:06.818717 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:41:07.012730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.012702 4990 generic.go:356] "Generic (PLEG): container finished" podID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerID="d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60" exitCode=0 Oct 30 19:41:07.012950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.012926 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerDied","Data":"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60"} Oct 30 19:41:07.013459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.013437 4990 scope.go:122] "RemoveContainer" containerID="d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60" Oct 30 19:41:07.015560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.015521 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:41:07.015660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.015615 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:41:07.015701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.015660 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:41:07.267175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.267138 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.518111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.518003 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.687905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.687855 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.776999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.776907 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:41:07.777237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.777190 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" containerID="cri-o://829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f" gracePeriod=30 Oct 30 19:41:07.921130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:07.919513 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.063856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:08.063478 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerStarted","Data":"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c"} Oct 30 19:41:08.112180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:08.112065 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.690114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:08.689162 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:41:08.690114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:08.689312 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:08.690114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:08.689370 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:09.174115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.172181 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:41:09.697118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.696642 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:09.697118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.696723 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:09.738497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.738460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.743312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.743281 4990 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:41:09.743458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.743340 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:41:09.895036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:09.894997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:10.684182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:10.684145 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:10.684796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:10.684762 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:10.979812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:10.979719 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:11.681115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:11.680501 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:11.681115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:11.680583 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:11.931170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:11.930788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.681119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:12.680441 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:12.681119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:12.680506 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:13.138354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.138324 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:41:13.147418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.147372 4990 generic.go:356] "Generic (PLEG): container finished" podID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerID="829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f" exitCode=0 Oct 30 19:41:13.147586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.147432 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" Oct 30 19:41:13.147586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.147442 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" event={"ID":"5f44ffe4-d341-4a70-a4a6-e37e8b270816","Type":"ContainerDied","Data":"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f"} Oct 30 19:41:13.147586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.147505 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq" event={"ID":"5f44ffe4-d341-4a70-a4a6-e37e8b270816","Type":"ContainerDied","Data":"7c5674ee9983cdf93aa7afaf93d93dae192929b94a123863d8e658b1b6ee05ac"} Oct 30 19:41:13.147586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.147536 4990 scope.go:122] "RemoveContainer" containerID="829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f" Oct 30 19:41:13.159123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.159069 4990 scope.go:122] "RemoveContainer" containerID="829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f" Oct 30 19:41:13.159480 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:13.159441 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f\": container with ID starting with 829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f not found: ID does not exist" containerID="829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f" Oct 30 19:41:13.159587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.159480 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f"} err="failed to get container status \"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f\": rpc error: code = NotFound desc = could not find container \"829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f\": container with ID starting with 829506b32652b43c5c83f1cba826149bdee2e98e8dc75c9eb66e83e11567394f not found: ID does not exist" Oct 30 19:41:13.262029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.261948 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262028 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262101 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262239 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262314 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262362 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262426 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262458 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.262602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262580 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\") pod \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\" (UID: \"5f44ffe4-d341-4a70-a4a6-e37e8b270816\") " Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262758 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.262874 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263086 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263265 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263360 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263383 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.263427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263398 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.263895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.263566 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.303156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.301277 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "kube-api-access-xkhrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:13.303156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.301378 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:13.308607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.308573 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.308725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.308636 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo" pod "5f44ffe4-d341-4a70-a4a6-e37e8b270816" (UID: "5f44ffe4-d341-4a70-a4a6-e37e8b270816"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:13.364948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364919 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.364948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364947 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.365197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364956 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.365197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364965 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.365197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364975 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhrz\" (UniqueName: \"kubernetes.io/projected/5f44ffe4-d341-4a70-a4a6-e37e8b270816-kube-api-access-xkhrz\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.365197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.364984 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f44ffe4-d341-4a70-a4a6-e37e8b270816-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.514816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.514723 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:41:13.517194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.517162 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:13.521454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.521424 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-4nbsq"] Oct 30 19:41:13.686113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.685426 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:41:13.686113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:13.685491 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 19:41:14.406368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:14.406331 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" path="/var/lib/kubelet/pods/5f44ffe4-d341-4a70-a4a6-e37e8b270816/volumes" Oct 30 19:41:14.684699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:14.684617 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:41:14.685657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:14.685635 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:41:15.157362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:15.157158 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.767720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:16.767682 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.939558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:16.939527 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:17.182250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:17.182209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.674431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:18.674391 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.863232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:18.863195 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.268620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:19.268583 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:41:19.303054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:19.303015 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:41:19.458380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:19.458344 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.621155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:19.621120 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.973476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:20.973440 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.230286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:23.230250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.581265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:23.581230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.269803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.269758 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" podUID="363207af-a81b-462b-9d87-7ffbf6e9fd63" containerName="payload-processing" containerID="cri-o://5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33" gracePeriod=25 Oct 30 19:41:24.303603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.303564 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" containerID="cri-o://5f6d787743d041f3fb2fa7855125f53c435da421cc5e69b147adac6494771cdf" gracePeriod=25 Oct 30 19:41:24.611428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.611402 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:41:24.628965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.628931 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\" (UniqueName: \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\") pod \"363207af-a81b-462b-9d87-7ffbf6e9fd63\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " Oct 30 19:41:24.629152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.629137 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\") pod \"363207af-a81b-462b-9d87-7ffbf6e9fd63\" (UID: \"363207af-a81b-462b-9d87-7ffbf6e9fd63\") " Oct 30 19:41:24.629478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.629438 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume" pod "363207af-a81b-462b-9d87-7ffbf6e9fd63" (UID: "363207af-a81b-462b-9d87-7ffbf6e9fd63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:24.629794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.629776 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363207af-a81b-462b-9d87-7ffbf6e9fd63-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:24.631379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.631350 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt" pod "363207af-a81b-462b-9d87-7ffbf6e9fd63" (UID: "363207af-a81b-462b-9d87-7ffbf6e9fd63"). InnerVolumeSpecName "kube-api-access-w8mlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:24.731401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:24.731367 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8mlt\" (UniqueName: \"kubernetes.io/projected/363207af-a81b-462b-9d87-7ffbf6e9fd63-kube-api-access-w8mlt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:25.263306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.263267 4990 generic.go:356] "Generic (PLEG): container finished" podID="363207af-a81b-462b-9d87-7ffbf6e9fd63" containerID="5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33" exitCode=0 Oct 30 19:41:25.263517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.263316 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" Oct 30 19:41:25.263517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.263351 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" event={"ID":"363207af-a81b-462b-9d87-7ffbf6e9fd63","Type":"ContainerDied","Data":"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33"} Oct 30 19:41:25.263517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.263410 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd" event={"ID":"363207af-a81b-462b-9d87-7ffbf6e9fd63","Type":"ContainerDied","Data":"41c7a57f00482cf6501c664941b53d2f94b9bc125d7f049cc7c2e438992d614a"} Oct 30 19:41:25.263517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.263432 4990 scope.go:122] "RemoveContainer" containerID="5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33" Oct 30 19:41:25.277470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.277254 4990 scope.go:122] "RemoveContainer" containerID="5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33" Oct 30 19:41:25.277753 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:25.277596 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33\": container with ID starting with 5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33 not found: ID does not exist" containerID="5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33" Oct 30 19:41:25.277753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.277630 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33"} err="failed to get container status \"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33\": rpc error: code = NotFound desc = could not find container \"5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33\": container with ID starting with 5645cc99d945c76e9ff4913e4a1f450025e92db9e7ce4de8d9a94cebc7f69b33 not found: ID does not exist" Oct 30 19:41:25.316768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.316734 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:41:25.322904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.322875 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8e6b67-865ddcdd9d-hh9pd"] Oct 30 19:41:25.422230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:25.416419 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.405532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:26.405495 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363207af-a81b-462b-9d87-7ffbf6e9fd63" path="/var/lib/kubelet/pods/363207af-a81b-462b-9d87-7ffbf6e9fd63/volumes" Oct 30 19:41:27.161769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:27.160659 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.710827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:28.710788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.869125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:28.869055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.037954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:29.037077 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.606111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:29.606062 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.765973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:29.765930 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:30.737748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:30.737494 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:31.710810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:31.710768 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:31.869078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:31.869043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:32.367629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:32.367538 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.583822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:33.583783 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:34.118829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:34.118788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.377841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:34.377756 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:41:34.566578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:34.566544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.700567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:34.700472 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:35.191129 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:35.191063 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:41:35.802454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:35.802412 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.035109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:36.035060 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.204893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:36.204814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.742941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:37.742902 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.378929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.378889 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" podUID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" containerName="payload-processing" containerID="cri-o://c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3" gracePeriod=25 Oct 30 19:41:39.668384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.668359 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:41:39.691154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.691124 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\" (UniqueName: \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\") pod \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " Oct 30 19:41:39.691410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.691390 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\") pod \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\" (UID: \"0a2e06e8-cf86-409f-b493-e261a7eb1f28\") " Oct 30 19:41:39.691782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.691754 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume" pod "0a2e06e8-cf86-409f-b493-e261a7eb1f28" (UID: "0a2e06e8-cf86-409f-b493-e261a7eb1f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:39.692083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.692063 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2e06e8-cf86-409f-b493-e261a7eb1f28-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:39.693339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.693318 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg" pod "0a2e06e8-cf86-409f-b493-e261a7eb1f28" (UID: "0a2e06e8-cf86-409f-b493-e261a7eb1f28"). InnerVolumeSpecName "kube-api-access-d27kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:39.750349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.750291 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.793478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:39.793442 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27kg\" (UniqueName: \"kubernetes.io/projected/0a2e06e8-cf86-409f-b493-e261a7eb1f28-kube-api-access-d27kg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:40.413730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.413696 4990 generic.go:356] "Generic (PLEG): container finished" podID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" containerID="c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3" exitCode=0 Oct 30 19:41:40.414134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.413759 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" Oct 30 19:41:40.414134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.413774 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" event={"ID":"0a2e06e8-cf86-409f-b493-e261a7eb1f28","Type":"ContainerDied","Data":"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3"} Oct 30 19:41:40.414134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.413829 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47" event={"ID":"0a2e06e8-cf86-409f-b493-e261a7eb1f28","Type":"ContainerDied","Data":"3c2b6b72fa93ab771d19055f19c4f03b681994ff752c3c609d7d8e5632d01069"} Oct 30 19:41:40.414134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.413850 4990 scope.go:122] "RemoveContainer" containerID="c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3" Oct 30 19:41:40.426657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.426641 4990 scope.go:122] "RemoveContainer" containerID="c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3" Oct 30 19:41:40.426973 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:40.426944 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3\": container with ID starting with c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3 not found: ID does not exist" containerID="c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3" Oct 30 19:41:40.427044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.426985 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3"} err="failed to get container status \"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3\": rpc error: code = NotFound desc = could not find container \"c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3\": container with ID starting with c0e2e453c24be69b6407703f575b20c552aefd2b30e2a0082a09e6d2425b9eb3 not found: ID does not exist" Oct 30 19:41:40.463129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.463083 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:41:40.472447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:40.472423 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c8a2a-7475f7cbd4-qzw47"] Oct 30 19:41:41.468276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:41.468235 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.080550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:42.080508 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.403595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:42.403519 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" path="/var/lib/kubelet/pods/0a2e06e8-cf86-409f-b493-e261a7eb1f28/volumes" Oct 30 19:41:43.220034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:43.220002 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.583589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:43.583547 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:44.453474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.453436 4990 generic.go:356] "Generic (PLEG): container finished" podID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerID="5f6d787743d041f3fb2fa7855125f53c435da421cc5e69b147adac6494771cdf" exitCode=0 Oct 30 19:41:44.453880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.453515 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" event={"ID":"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa","Type":"ContainerDied","Data":"5f6d787743d041f3fb2fa7855125f53c435da421cc5e69b147adac6494771cdf"} Oct 30 19:41:44.558930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.558878 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:41:44.561055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.560957 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" podUID="30682551-dd30-448a-930b-263880628b44" containerName="maas-api" containerID="cri-o://c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28" gracePeriod=30 Oct 30 19:41:44.657027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.656996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:44.713271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.713243 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:41:44.782750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.778729 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\" (UniqueName: \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\") pod \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " Oct 30 19:41:44.782750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.778786 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\") pod \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\" (UID: \"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa\") " Oct 30 19:41:44.782750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.779901 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume" pod "b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" (UID: "b2adf0da-4ba2-4e31-b6b7-e71ee01909aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:44.844165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.834423 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd" pod "b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" (UID: "b2adf0da-4ba2-4e31-b6b7-e71ee01909aa"). InnerVolumeSpecName "kube-api-access-gqgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:44.903122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.892455 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgzd\" (UniqueName: \"kubernetes.io/projected/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-kube-api-access-gqgzd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:44.903122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:44.892493 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:45.084179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.084135 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:41:45.199912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.199810 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\") pod \"30682551-dd30-448a-930b-263880628b44\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " Oct 30 19:41:45.199912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.199858 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\" (UniqueName: \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\") pod \"30682551-dd30-448a-930b-263880628b44\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " Oct 30 19:41:45.199912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.199903 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\") pod \"30682551-dd30-448a-930b-263880628b44\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " Oct 30 19:41:45.200255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.199941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\") pod \"30682551-dd30-448a-930b-263880628b44\" (UID: \"30682551-dd30-448a-930b-263880628b44\") " Oct 30 19:41:45.200717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.200681 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca" pod "30682551-dd30-448a-930b-263880628b44" (UID: "30682551-dd30-448a-930b-263880628b44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:45.208529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.208497 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs" pod "30682551-dd30-448a-930b-263880628b44" (UID: "30682551-dd30-448a-930b-263880628b44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:45.208666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.208554 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2" pod "30682551-dd30-448a-930b-263880628b44" (UID: "30682551-dd30-448a-930b-263880628b44"). InnerVolumeSpecName "kube-api-access-2llk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:45.210064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.210036 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls" pod "30682551-dd30-448a-930b-263880628b44" (UID: "30682551-dd30-448a-930b-263880628b44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:45.301795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.301756 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:45.301795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.301794 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llk2\" (UniqueName: \"kubernetes.io/projected/30682551-dd30-448a-930b-263880628b44-kube-api-access-2llk2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:45.302021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.301809 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30682551-dd30-448a-930b-263880628b44-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:45.302021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.301824 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30682551-dd30-448a-930b-263880628b44-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:45.464246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.464160 4990 generic.go:356] "Generic (PLEG): container finished" podID="30682551-dd30-448a-930b-263880628b44" containerID="c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28" exitCode=0 Oct 30 19:41:45.464246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.464209 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" Oct 30 19:41:45.464718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.464241 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" event={"ID":"30682551-dd30-448a-930b-263880628b44","Type":"ContainerDied","Data":"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28"} Oct 30 19:41:45.464718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.464304 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw" event={"ID":"30682551-dd30-448a-930b-263880628b44","Type":"ContainerDied","Data":"4ebe79638b4951f4a9d81daf3d4a3faa04f7ef98d326a38fb16fa5005c825ef5"} Oct 30 19:41:45.464718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.464334 4990 scope.go:122] "RemoveContainer" containerID="c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28" Oct 30 19:41:45.466184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.466153 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" event={"ID":"b2adf0da-4ba2-4e31-b6b7-e71ee01909aa","Type":"ContainerDied","Data":"8173f90fe8e74d838ea94a68de451663879a3ea389ecdb6efb11ac3c1b363581"} Oct 30 19:41:45.466184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.466166 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths" Oct 30 19:41:45.478408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.478391 4990 scope.go:122] "RemoveContainer" containerID="c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28" Oct 30 19:41:45.478838 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:41:45.478815 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28\": container with ID starting with c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28 not found: ID does not exist" containerID="c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28" Oct 30 19:41:45.478948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.478847 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28"} err="failed to get container status \"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28\": rpc error: code = NotFound desc = could not find container \"c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28\": container with ID starting with c34e8f99b65bed6b8647d2b4a7301aeb1a1a0be6db609724d479d9bc1f78dc28 not found: ID does not exist" Oct 30 19:41:45.478948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.478869 4990 scope.go:122] "RemoveContainer" containerID="5f6d787743d041f3fb2fa7855125f53c435da421cc5e69b147adac6494771cdf" Oct 30 19:41:45.521008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.520980 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:41:45.525601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.525580 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8e6b67-68584d454c-pzths"] Oct 30 19:41:45.564883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.564855 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:41:45.575672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.575643 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a7c3b0-75dc59cd75-m6fgw"] Oct 30 19:41:45.629332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.629284 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.810877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.810842 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.958707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:45.958651 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.329631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:46.329556 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.406741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:46.406701 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30682551-dd30-448a-930b-263880628b44" path="/var/lib/kubelet/pods/30682551-dd30-448a-930b-263880628b44/volumes" Oct 30 19:41:46.407444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:46.407423 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" path="/var/lib/kubelet/pods/b2adf0da-4ba2-4e31-b6b7-e71ee01909aa/volumes" Oct 30 19:41:47.319007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:47.318954 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:47.731845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:47.730852 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:49.512198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:49.512159 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.627131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:50.627084 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.934306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:50.934208 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.355380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.355337 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:41:51.355620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.355595 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" podUID="7ea9cc02-0ed9-43d5-902b-702155afd366" containerName="maas-api" containerID="cri-o://5e73a14957bd83225c4acb8777f8742fae9c1ebc6e3c576a4b032f99797adfc5" gracePeriod=30 Oct 30 19:41:51.470496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.470459 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.534790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.534727 4990 generic.go:356] "Generic (PLEG): container finished" podID="7ea9cc02-0ed9-43d5-902b-702155afd366" containerID="5e73a14957bd83225c4acb8777f8742fae9c1ebc6e3c576a4b032f99797adfc5" exitCode=0 Oct 30 19:41:51.535051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.535024 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" event={"ID":"7ea9cc02-0ed9-43d5-902b-702155afd366","Type":"ContainerDied","Data":"5e73a14957bd83225c4acb8777f8742fae9c1ebc6e3c576a4b032f99797adfc5"} Oct 30 19:41:51.656742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.656715 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:41:51.715390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.715357 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") pod \"7ea9cc02-0ed9-43d5-902b-702155afd366\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " Oct 30 19:41:51.715599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.715429 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\") pod \"7ea9cc02-0ed9-43d5-902b-702155afd366\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " Oct 30 19:41:51.715599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.715489 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\" (UniqueName: \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\") pod \"7ea9cc02-0ed9-43d5-902b-702155afd366\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " Oct 30 19:41:51.715726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.715595 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\") pod \"7ea9cc02-0ed9-43d5-902b-702155afd366\" (UID: \"7ea9cc02-0ed9-43d5-902b-702155afd366\") " Oct 30 19:41:51.715965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.715913 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca" pod "7ea9cc02-0ed9-43d5-902b-702155afd366" (UID: "7ea9cc02-0ed9-43d5-902b-702155afd366"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:51.716698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.716635 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea9cc02-0ed9-43d5-902b-702155afd366-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.717918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.717889 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4" pod "7ea9cc02-0ed9-43d5-902b-702155afd366" (UID: "7ea9cc02-0ed9-43d5-902b-702155afd366"). InnerVolumeSpecName "kube-api-access-zwvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:51.718019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.717976 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls" pod "7ea9cc02-0ed9-43d5-902b-702155afd366" (UID: "7ea9cc02-0ed9-43d5-902b-702155afd366"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:51.718352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.718331 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs" pod "7ea9cc02-0ed9-43d5-902b-702155afd366" (UID: "7ea9cc02-0ed9-43d5-902b-702155afd366"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:51.748675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.748641 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.818062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.818013 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.818062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.818057 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea9cc02-0ed9-43d5-902b-702155afd366-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.818294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.818073 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvb4\" (UniqueName: \"kubernetes.io/projected/7ea9cc02-0ed9-43d5-902b-702155afd366-kube-api-access-zwvb4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.926954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:51.918623 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.271007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.270908 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.419757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.419719 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.545580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.545423 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" Oct 30 19:41:52.545734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.545444 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99" event={"ID":"7ea9cc02-0ed9-43d5-902b-702155afd366","Type":"ContainerDied","Data":"afd8bc4fae69cce07c7dadb70c6a173e24121094d60701557685b1b94c034331"} Oct 30 19:41:52.545734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.545722 4990 scope.go:122] "RemoveContainer" containerID="5e73a14957bd83225c4acb8777f8742fae9c1ebc6e3c576a4b032f99797adfc5" Oct 30 19:41:52.562818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.562782 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.598047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.598009 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:41:52.605833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.605803 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c8a2a-6c5f84bd99-8bn99"] Oct 30 19:41:52.839612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:52.839574 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.050736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:53.050693 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.375123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:53.374173 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.035986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:54.035937 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.404585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:54.404548 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea9cc02-0ed9-43d5-902b-702155afd366" path="/var/lib/kubelet/pods/7ea9cc02-0ed9-43d5-902b-702155afd366/volumes" Oct 30 19:41:54.549661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:54.549625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.720746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:54.720608 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.282610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:55.282575 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.043267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:57.043232 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.149147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:58.149064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.313536 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:58.313494 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.057767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:59.057732 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.927795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:41:59.927754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.101683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.101635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.215041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.214930 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:42:00.216042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.216009 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" containerID="cri-o://649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978" gracePeriod=30 Oct 30 19:42:00.281963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.281933 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.484208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.484081 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.874374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:00.874340 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.051805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:01.051767 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:42:01.052343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:01.052023 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" containerID="cri-o://decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466" gracePeriod=30 Oct 30 19:42:01.059453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:01.059424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.234463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:01.234366 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.397116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:01.385991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.333228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:03.332802 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.020581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.020541 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.107776 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:05.107735 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:42:05.519604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.519571 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:42:05.623870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.623768 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.623870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.623834 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.623963 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.623994 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.624040 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.624109 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.624171 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.624317 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.624470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.624361 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\") pod \"06c0b71e-6645-49b2-a868-c26fee1799ec\" (UID: \"06c0b71e-6645-49b2-a868-c26fee1799ec\") " Oct 30 19:42:05.626648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.626212 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:05.626648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.626611 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:05.627253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.627158 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:05.627253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.627230 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:05.627487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.627466 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:05.629987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.629952 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:05.631347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.631316 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "kube-api-access-vlk8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:05.631580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.631555 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:05.633353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.633323 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo" pod "06c0b71e-6645-49b2-a868-c26fee1799ec" (UID: "06c0b71e-6645-49b2-a868-c26fee1799ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:05.685070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.685032 4990 generic.go:356] "Generic (PLEG): container finished" podID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerID="649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978" exitCode=0 Oct 30 19:42:05.685607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.685376 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" Oct 30 19:42:05.690490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.685399 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerDied","Data":"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978"} Oct 30 19:42:05.690490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.689517 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5" event={"ID":"06c0b71e-6645-49b2-a868-c26fee1799ec","Type":"ContainerDied","Data":"f0083f974df5a073eb09f879c29f238ab838992a6d00391f30c3375db12e1eed"} Oct 30 19:42:05.690490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.689567 4990 scope.go:122] "RemoveContainer" containerID="649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978" Oct 30 19:42:05.708464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.708444 4990 scope.go:122] "RemoveContainer" containerID="2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1" Oct 30 19:42:05.719954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.719934 4990 scope.go:122] "RemoveContainer" containerID="649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978" Oct 30 19:42:05.720243 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:05.720223 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978\": container with ID starting with 649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978 not found: ID does not exist" containerID="649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978" Oct 30 19:42:05.720334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.720250 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978"} err="failed to get container status \"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978\": rpc error: code = NotFound desc = could not find container \"649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978\": container with ID starting with 649f36a9e692539499bf983ff3c206bee81877560083b9b5a92706587b570978 not found: ID does not exist" Oct 30 19:42:05.720334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.720265 4990 scope.go:122] "RemoveContainer" containerID="2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1" Oct 30 19:42:05.720504 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:05.720480 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1\": container with ID starting with 2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1 not found: ID does not exist" containerID="2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1" Oct 30 19:42:05.720565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.720508 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1"} err="failed to get container status \"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1\": rpc error: code = NotFound desc = could not find container \"2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1\": container with ID starting with 2f15936c9a5f8e4236503da4b082a450d0b1a84a346963703bf14dc749a4a2c1 not found: ID does not exist" Oct 30 19:42:05.742246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742199 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlk8k\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-kube-api-access-vlk8k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742226 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742243 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742255 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742263 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742272 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742280 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c0b71e-6645-49b2-a868-c26fee1799ec-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742288 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.742589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.742296 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c0b71e-6645-49b2-a868-c26fee1799ec-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:05.743736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.743711 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:42:05.747896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:05.747860 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8e6b67-openshift-default-5c89b78cd5-786g5"] Oct 30 19:42:06.344745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.344714 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:42:06.407724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.407681 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" path="/var/lib/kubelet/pods/06c0b71e-6645-49b2-a868-c26fee1799ec/volumes" Oct 30 19:42:06.453042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453006 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453141 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453179 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453220 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453288 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453327 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.453458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453339 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453414 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453604 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453651 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\") pod \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\" (UID: \"8c58efb1-eea9-40e8-9951-5b8e8b7ba230\") " Oct 30 19:42:06.453709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453656 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:06.453931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453910 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.454018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.453982 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.454216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454197 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.454582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454530 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.454706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454624 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.454706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454643 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.454706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454657 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.454706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.454672 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.456801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.456773 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "kube-api-access-4qspq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.460878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.460853 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:06.462606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.462367 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.462814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.462789 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token" pod "8c58efb1-eea9-40e8-9951-5b8e8b7ba230" (UID: "8c58efb1-eea9-40e8-9951-5b8e8b7ba230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.556576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.556541 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qspq\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-kube-api-access-4qspq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.556576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.556573 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.556576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.556584 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.556799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.556592 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c58efb1-eea9-40e8-9951-5b8e8b7ba230-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.697040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.696950 4990 generic.go:356] "Generic (PLEG): container finished" podID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerID="decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466" exitCode=0 Oct 30 19:42:06.697040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.697002 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" Oct 30 19:42:06.697279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.697033 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" event={"ID":"8c58efb1-eea9-40e8-9951-5b8e8b7ba230","Type":"ContainerDied","Data":"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466"} Oct 30 19:42:06.697279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.697101 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j" event={"ID":"8c58efb1-eea9-40e8-9951-5b8e8b7ba230","Type":"ContainerDied","Data":"f196fe4670bda5dbd14420a75cff7d4a50a749c5787a4e872c0d31f97c080aad"} Oct 30 19:42:06.697279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.697115 4990 scope.go:122] "RemoveContainer" containerID="decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466" Oct 30 19:42:06.709976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.709956 4990 scope.go:122] "RemoveContainer" containerID="decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466" Oct 30 19:42:06.710269 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:06.710248 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466\": container with ID starting with decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466 not found: ID does not exist" containerID="decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466" Oct 30 19:42:06.710353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.710276 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466"} err="failed to get container status \"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466\": rpc error: code = NotFound desc = could not find container \"decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466\": container with ID starting with decc20f3c6b99af99597813f04d9fd02031d52bbfeb1a27b9d5edad6644ec466 not found: ID does not exist" Oct 30 19:42:06.744000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.743969 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:42:06.753223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.753186 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-wrz4j"] Oct 30 19:42:06.939428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:06.939387 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:08.403458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:08.403423 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" path="/var/lib/kubelet/pods/8c58efb1-eea9-40e8-9951-5b8e8b7ba230/volumes" Oct 30 19:42:08.732198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:08.727875 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.407587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:10.407545 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.778792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:10.778695 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.538836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.538798 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm"] Oct 30 19:42:11.539534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539513 4990 cpu_manager.go:414] "containerMap: removing container" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:11.539534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539529 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539553 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539559 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539566 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539572 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539587 4990 cpu_manager.go:414] "containerMap: removing container" podUID="30682551-dd30-448a-930b-263880628b44" containerName="maas-api" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539591 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30682551-dd30-448a-930b-263880628b44" containerName="maas-api" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539603 4990 cpu_manager.go:414] "containerMap: removing container" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539609 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539618 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" containerName="payload-processing" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539622 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" containerName="payload-processing" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539634 4990 cpu_manager.go:414] "containerMap: removing container" podUID="363207af-a81b-462b-9d87-7ffbf6e9fd63" containerName="payload-processing" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539638 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363207af-a81b-462b-9d87-7ffbf6e9fd63" containerName="payload-processing" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539644 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7ea9cc02-0ed9-43d5-902b-702155afd366" containerName="maas-api" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539648 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea9cc02-0ed9-43d5-902b-702155afd366" containerName="maas-api" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539669 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" Oct 30 19:42:11.539697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539673 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539767 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2adf0da-4ba2-4e31-b6b7-e71ee01909aa" containerName="payload-pre-processing" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539776 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539784 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f44ffe4-d341-4a70-a4a6-e37e8b270816" containerName="istio-proxy" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539790 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c58efb1-eea9-40e8-9951-5b8e8b7ba230" containerName="istio-proxy" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539797 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="30682551-dd30-448a-930b-263880628b44" containerName="maas-api" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539804 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2e06e8-cf86-409f-b493-e261a7eb1f28" containerName="payload-processing" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539812 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="363207af-a81b-462b-9d87-7ffbf6e9fd63" containerName="payload-processing" Oct 30 19:42:11.540204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.539816 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea9cc02-0ed9-43d5-902b-702155afd366" containerName="maas-api" Oct 30 19:42:11.543573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.543555 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.555900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.555872 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:11.556556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.556531 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.556649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.556581 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6plw8\" (UniqueName: \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.556703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.556689 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.566735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.565843 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:42:11.591143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.572199 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm"] Oct 30 19:42:11.658424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.658390 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.658594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.658439 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6plw8\" (UniqueName: \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.658594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.658493 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.659174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.659147 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.660937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.660913 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.675543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.675512 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6plw8\" (UniqueName: \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\") pod \"maas-api-revoke-keys-e2e-inf-b-19f646-fgffm\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:11.856452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:11.856416 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:12.171538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:12.171513 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm"] Oct 30 19:42:12.174532 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:42:12.174500 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d3b6a8_40d4_404d_8f25_e5091de1de15.slice/crio-05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13 WatchSource:0}: Error finding container 05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13: Status 404 returned error can't find the container with id 05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13 Oct 30 19:42:12.347325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:12.347294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.749798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:12.749694 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" event={"ID":"d5d3b6a8-40d4-404d-8f25-e5091de1de15","Type":"ContainerStarted","Data":"4b4849dfe8781ffaf3f7e902338e36d8369adfdc25b3330cfe045699bc08f2f1"} Oct 30 19:42:12.749798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:12.749758 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" event={"ID":"d5d3b6a8-40d4-404d-8f25-e5091de1de15","Type":"ContainerStarted","Data":"05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13"} Oct 30 19:42:12.786794 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:12.786740 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" podStartSLOduration=1.786726341 podStartE2EDuration="1.786726341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:12.783629559 +0000 UTC m=+2240.495154927" watchObservedRunningTime="2026-10-30 19:42:12.786726341 +0000 UTC m=+2240.498251706" Oct 30 19:42:13.759772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:13.759736 4990 generic.go:356] "Generic (PLEG): container finished" podID="d5d3b6a8-40d4-404d-8f25-e5091de1de15" containerID="4b4849dfe8781ffaf3f7e902338e36d8369adfdc25b3330cfe045699bc08f2f1" exitCode=0 Oct 30 19:42:13.760183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:13.759813 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" event={"ID":"d5d3b6a8-40d4-404d-8f25-e5091de1de15","Type":"ContainerDied","Data":"4b4849dfe8781ffaf3f7e902338e36d8369adfdc25b3330cfe045699bc08f2f1"} Oct 30 19:42:13.810882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:13.810849 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:14.407642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:14.407607 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:14.954312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:14.954289 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:15.011652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.011611 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\") pod \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " Oct 30 19:42:15.011852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.011820 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\") pod \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " Oct 30 19:42:15.012040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.012012 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\" (UniqueName: \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\") pod \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\" (UID: \"d5d3b6a8-40d4-404d-8f25-e5091de1de15\") " Oct 30 19:42:15.012434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.012409 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca" pod "d5d3b6a8-40d4-404d-8f25-e5091de1de15" (UID: "d5d3b6a8-40d4-404d-8f25-e5091de1de15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:15.012798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.012778 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:15.014973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.014949 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca" pod "d5d3b6a8-40d4-404d-8f25-e5091de1de15" (UID: "d5d3b6a8-40d4-404d-8f25-e5091de1de15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:15.015141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.015075 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8" pod "d5d3b6a8-40d4-404d-8f25-e5091de1de15" (UID: "d5d3b6a8-40d4-404d-8f25-e5091de1de15"). InnerVolumeSpecName "kube-api-access-6plw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:15.114741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.114700 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d3b6a8-40d4-404d-8f25-e5091de1de15-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:15.114741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.114741 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6plw8\" (UniqueName: \"kubernetes.io/projected/d5d3b6a8-40d4-404d-8f25-e5091de1de15-kube-api-access-6plw8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:15.778637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.778608 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" Oct 30 19:42:15.778825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.778602 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm" event={"ID":"d5d3b6a8-40d4-404d-8f25-e5091de1de15","Type":"ContainerDied","Data":"05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13"} Oct 30 19:42:15.778825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:15.778738 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d197da610fa32d2560739aab6da4f6a146c5b3029e27af2c985f2aff454a13" Oct 30 19:42:16.437867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:16.437815 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.222731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:18.222684 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.871128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:19.871061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.300942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:20.300906 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:21.949573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:21.949535 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.059055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:22.059018 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:42:22.223268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:22.223188 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.373316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:22.373275 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.753235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:23.753173 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.018718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.018634 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:42:25.020381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.020349 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d5d3b6a8-40d4-404d-8f25-e5091de1de15" containerName="revoke-keys" Oct 30 19:42:25.020381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.020379 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d3b6a8-40d4-404d-8f25-e5091de1de15" containerName="revoke-keys" Oct 30 19:42:25.020599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.020583 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d3b6a8-40d4-404d-8f25-e5091de1de15" containerName="revoke-keys" Oct 30 19:42:25.020652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.020615 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c0b71e-6645-49b2-a868-c26fee1799ec" containerName="istio-proxy" Oct 30 19:42:25.023555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.023535 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.042316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.042254 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:42:25.044712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.044687 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"default-dockercfg-d5dwr\"" Oct 30 19:42:25.045163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.044810 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"openshift-service-ca.crt\"" Oct 30 19:42:25.055401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.055285 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"auth-test-model-6b7ccc-kserve-self-signed-certs\"" Oct 30 19:42:25.055401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.055374 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"kube-root-ca.crt\"" Oct 30 19:42:25.087953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.087913 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.088174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.088153 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85p72\" (UniqueName: \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.088237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.088212 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.088344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.088313 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.088497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.088388 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.088497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.088464 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.123772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.123718 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.189981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.189942 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190113 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85p72\" (UniqueName: \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190174 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190207 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190264 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190356 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190702 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190744 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.190829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.190757 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.192504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.192482 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.192610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.192590 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.207157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.207136 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85p72\" (UniqueName: \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\") pod \"auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.356877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.356835 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:25.558059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.558018 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.698997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.698969 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:42:25.700697 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:42:25.700664 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21221a17_6537_473a_86b5_28f1786f7271.slice/crio-a5394deb5a64e63d83d460b9306efabbf6665b914bc22460cfe779d4b01b2382 WatchSource:0}: Error finding container a5394deb5a64e63d83d460b9306efabbf6665b914bc22460cfe779d4b01b2382: Status 404 returned error can't find the container with id a5394deb5a64e63d83d460b9306efabbf6665b914bc22460cfe779d4b01b2382 Oct 30 19:42:25.898876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.898839 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" event={"ID":"21221a17-6537-473a-86b5-28f1786f7271","Type":"ContainerStarted","Data":"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025"} Oct 30 19:42:25.898876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.898888 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" event={"ID":"21221a17-6537-473a-86b5-28f1786f7271","Type":"ContainerStarted","Data":"a5394deb5a64e63d83d460b9306efabbf6665b914bc22460cfe779d4b01b2382"} Oct 30 19:42:25.934904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:25.934851 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" podStartSLOduration=1.9348360690000002 podStartE2EDuration="1.934836069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:25.933957003 +0000 UTC m=+2253.645482369" watchObservedRunningTime="2026-10-30 19:42:25.934836069 +0000 UTC m=+2253.646361436" Oct 30 19:42:26.356978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:26.356946 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:27.060015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:27.059974 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" containerID="cri-o://6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd" gracePeriod=25 Oct 30 19:42:27.142557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:27.142513 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.812384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:28.812345 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:28.978714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:28.978668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:29.347185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:29.347147 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:30.452872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:30.452832 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:32.143406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:32.143367 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:33.927105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:33.927046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.099510 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:35.099470 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:42:35.357132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:35.357031 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:35.372009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:35.371969 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:35.452496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:35.452465 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.997733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:35.997705 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:42:37.080930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:37.080894 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:38.578508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.578470 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:42:38.583381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.583362 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.608281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.608256 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f6cef-serving-cert\"" Oct 30 19:42:38.615497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.615463 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:42:38.694000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.693957 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw575\" (UniqueName: \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.694199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.694032 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.694372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.694346 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.694509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.694487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.709990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.709958 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:38.795861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.795821 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.796255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.796223 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw575\" (UniqueName: \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.796401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.796290 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.797552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.796523 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.797801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.797102 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.797930 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:38.797484 4990 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-serving-cert: secret "maas-api-e2e-inf-a-1f6cef-serving-cert" not found Oct 30 19:42:38.798109 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:38.798077 4990 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls podName:25cc5e20-2452-4799-937e-bd2c0c7a339c nodeName:}" failed. No retries permitted until 2026-10-30 19:42:39.298054088 +0000 UTC m=+2267.009579447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls") pod "maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" (UID: "25cc5e20-2452-4799-937e-bd2c0c7a339c") : secret "maas-api-e2e-inf-a-1f6cef-serving-cert" not found Oct 30 19:42:38.802589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.802566 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.817818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.817786 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw575\" (UniqueName: \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:38.851329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:38.851234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.092245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.092205 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.305211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.305170 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:39.307773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.307743 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:39.346553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.346520 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:39.499279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.499245 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:39.810260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:39.810231 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:42:39.811753 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:42:39.811677 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cc5e20_2452_4799_937e_bd2c0c7a339c.slice/crio-16f530d5105ed22847246bb4790feffd7ea6c13541cbf7c62ecfadfca41237fd WatchSource:0}: Error finding container 16f530d5105ed22847246bb4790feffd7ea6c13541cbf7c62ecfadfca41237fd: Status 404 returned error can't find the container with id 16f530d5105ed22847246bb4790feffd7ea6c13541cbf7c62ecfadfca41237fd Oct 30 19:42:40.020261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:40.020223 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" event={"ID":"25cc5e20-2452-4799-937e-bd2c0c7a339c","Type":"ContainerStarted","Data":"16f530d5105ed22847246bb4790feffd7ea6c13541cbf7c62ecfadfca41237fd"} Oct 30 19:42:40.528006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:40.526485 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:41.039312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:41.039270 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" event={"ID":"25cc5e20-2452-4799-937e-bd2c0c7a339c","Type":"ContainerStarted","Data":"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c"} Oct 30 19:42:41.039723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:41.039390 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:41.076228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:41.076171 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" podStartSLOduration=2.660519841 podStartE2EDuration="3.076154398s" totalImagesPullingTime="415.634557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:41.073123724 +0000 UTC m=+2268.784649084" watchObservedRunningTime="2026-10-30 19:42:41.076154398 +0000 UTC m=+2268.787679765" Oct 30 19:42:41.316084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:41.316046 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:42:41.316294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:41.316270 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" containerID="cri-o://b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d" gracePeriod=30 Oct 30 19:42:42.197658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:42.197619 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.213682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:44.213645 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.360696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.360664 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:42:45.372382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.372344 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.395207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.395174 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:42:45.402628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.402592 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-614c02\"/\"default-dockercfg-mkvsj\"" Oct 30 19:42:45.402782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.402670 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-614c02\"/\"openshift-service-ca.crt\"" Oct 30 19:42:45.402851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.402827 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-614c02\"/\"test-model-614c02-kserve-self-signed-certs\"" Oct 30 19:42:45.402913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.402871 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-614c02\"/\"kube-root-ca.crt\"" Oct 30 19:42:45.435871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.435835 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.458699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.458669 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5g82\" (UniqueName: \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.458869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.458824 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.458942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.458878 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.459002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.458985 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.459060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.459014 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.459688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.459662 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561017 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5g82\" (UniqueName: \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561162 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561199 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561266 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561282 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561435 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561660 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561708 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.561821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.561793 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.563767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.563742 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.563985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.563960 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.581270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.581247 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5g82\" (UniqueName: \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\") pod \"test-model-614c02-kserve-8c5d7c9b7-tbzpr\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:45.699975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:45.699874 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:46.037163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.025313 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:42:46.131637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.131590 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" event={"ID":"9c74e543-3323-481f-b645-78bcb707f3e6","Type":"ContainerStarted","Data":"5b16903527337d1f88f8e191b11bc9d2c91717fdabdec7319db1e855e4bb7f33"} Oct 30 19:42:46.485701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.485658 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:46.651698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.651661 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:42:46.686776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686740 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.686959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686831 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.686959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686878 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.686959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686907 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.686959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686936 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.687277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.686983 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.687277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687051 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:46.687277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687109 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.687277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687257 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.687501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687308 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:46.687501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687348 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:46.687501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687376 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\") pod \"72dec97f-23a4-47bd-980c-11fab281d90c\" (UID: \"72dec97f-23a4-47bd-980c-11fab281d90c\") " Oct 30 19:42:46.687676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687618 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:46.687676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.687638 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:46.688037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.688014 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.688184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.688039 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.688184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.688050 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72dec97f-23a4-47bd-980c-11fab281d90c-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.688184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.688058 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.688184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.688066 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.689763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.689726 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:46.689913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.689891 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:46.690213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.690192 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:46.691532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.691507 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5" pod "72dec97f-23a4-47bd-980c-11fab281d90c" (UID: "72dec97f-23a4-47bd-980c-11fab281d90c"). InnerVolumeSpecName "kube-api-access-84rz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:46.789957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.789915 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rz5\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-kube-api-access-84rz5\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.789957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.789954 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72dec97f-23a4-47bd-980c-11fab281d90c-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.790190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.789969 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72dec97f-23a4-47bd-980c-11fab281d90c-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:46.790190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:46.789981 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72dec97f-23a4-47bd-980c-11fab281d90c-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:47.055874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.055804 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:42:47.147319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.147285 4990 generic.go:356] "Generic (PLEG): container finished" podID="72dec97f-23a4-47bd-980c-11fab281d90c" containerID="b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d" exitCode=0 Oct 30 19:42:47.147521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.147465 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" Oct 30 19:42:47.148244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.148213 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerDied","Data":"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d"} Oct 30 19:42:47.148486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.148450 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst" event={"ID":"72dec97f-23a4-47bd-980c-11fab281d90c","Type":"ContainerDied","Data":"129309e821da3d3b624adefb1127ce307e39dfa03c595f4575a2133de71fef4b"} Oct 30 19:42:47.148950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.148927 4990 scope.go:122] "RemoveContainer" containerID="b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d" Oct 30 19:42:47.151841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.150970 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" event={"ID":"9c74e543-3323-481f-b645-78bcb707f3e6","Type":"ContainerStarted","Data":"8fccc4b433775bc46db2b7382f054941b92123f8b1e0c216dee526f2a12656d4"} Oct 30 19:42:47.177392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.177282 4990 scope.go:122] "RemoveContainer" containerID="496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a" Oct 30 19:42:47.191916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.191826 4990 scope.go:122] "RemoveContainer" containerID="b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d" Oct 30 19:42:47.192202 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:47.192179 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d\": container with ID starting with b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d not found: ID does not exist" containerID="b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d" Oct 30 19:42:47.192302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.192213 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d"} err="failed to get container status \"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d\": rpc error: code = NotFound desc = could not find container \"b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d\": container with ID starting with b2078772a4621dc4e59dc9d5456f83d9657f8c543821a44a2e68739809dd415d not found: ID does not exist" Oct 30 19:42:47.192302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.192238 4990 scope.go:122] "RemoveContainer" containerID="496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a" Oct 30 19:42:47.192522 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:47.192497 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a\": container with ID starting with 496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a not found: ID does not exist" containerID="496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a" Oct 30 19:42:47.192566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.192534 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a"} err="failed to get container status \"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a\": rpc error: code = NotFound desc = could not find container \"496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a\": container with ID starting with 496a485f7cf44c5bfdb1e21d7d6beddefc46ef05fbc923179ebce08216ffe65a not found: ID does not exist" Oct 30 19:42:47.216887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.216847 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:42:47.225461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.225417 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-b5wst"] Oct 30 19:42:47.269382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.268360 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" podStartSLOduration=2.268340874 podStartE2EDuration="2.268340874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:47.262144782 +0000 UTC m=+2274.973670145" watchObservedRunningTime="2026-10-30 19:42:47.268340874 +0000 UTC m=+2274.979866241" Oct 30 19:42:47.355185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.355157 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:42:47.401990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.401958 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\" (UniqueName: \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\") pod \"3522964f-5fe5-432b-a4ab-7b8e8756d435\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " Oct 30 19:42:47.402207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.402108 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\") pod \"3522964f-5fe5-432b-a4ab-7b8e8756d435\" (UID: \"3522964f-5fe5-432b-a4ab-7b8e8756d435\") " Oct 30 19:42:47.402990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.402946 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume" pod "3522964f-5fe5-432b-a4ab-7b8e8756d435" (UID: "3522964f-5fe5-432b-a4ab-7b8e8756d435"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:47.407155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.405496 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng" pod "3522964f-5fe5-432b-a4ab-7b8e8756d435" (UID: "3522964f-5fe5-432b-a4ab-7b8e8756d435"). InnerVolumeSpecName "kube-api-access-k66ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:47.504785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.504738 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3522964f-5fe5-432b-a4ab-7b8e8756d435-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:47.504785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.504774 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k66ng\" (UniqueName: \"kubernetes.io/projected/3522964f-5fe5-432b-a4ab-7b8e8756d435-kube-api-access-k66ng\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:42:47.700830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:47.700752 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:48.160831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.160792 4990 generic.go:356] "Generic (PLEG): container finished" podID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerID="6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd" exitCode=0 Oct 30 19:42:48.161004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.160839 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" Oct 30 19:42:48.161004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.160875 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" event={"ID":"3522964f-5fe5-432b-a4ab-7b8e8756d435","Type":"ContainerDied","Data":"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd"} Oct 30 19:42:48.161004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.160938 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8" event={"ID":"3522964f-5fe5-432b-a4ab-7b8e8756d435","Type":"ContainerDied","Data":"546574d8b17213512021f8595e27be4130f01a8abb655e73c9b18b6009dd0f5a"} Oct 30 19:42:48.161004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.160965 4990 scope.go:122] "RemoveContainer" containerID="6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd" Oct 30 19:42:48.173161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.173143 4990 scope.go:122] "RemoveContainer" containerID="6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd" Oct 30 19:42:48.173422 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:42:48.173400 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd\": container with ID starting with 6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd not found: ID does not exist" containerID="6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd" Oct 30 19:42:48.173560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.173433 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd"} err="failed to get container status \"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd\": rpc error: code = NotFound desc = could not find container \"6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd\": container with ID starting with 6d34bb47c250130bb13daffbb5df994657cd688ef0fa1d7803eb5c379595e8cd not found: ID does not exist" Oct 30 19:42:48.214087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.214024 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:42:48.219678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.219652 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f646-779d5cff99-s9zt8"] Oct 30 19:42:48.405608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.405549 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" path="/var/lib/kubelet/pods/3522964f-5fe5-432b-a4ab-7b8e8756d435/volumes" Oct 30 19:42:48.406298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.406275 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" path="/var/lib/kubelet/pods/72dec97f-23a4-47bd-980c-11fab281d90c/volumes" Oct 30 19:42:48.485170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:48.485012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:49.155127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:49.155071 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.082961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:50.082921 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.754324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:50.754288 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.926705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:50.926672 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm"] Oct 30 19:42:50.933935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:50.933904 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.935438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:50.935418 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f646-fgffm"] Oct 30 19:42:51.142267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:51.142236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:51.463461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:51.463369 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:52.416708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:52.416669 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d3b6a8-40d4-404d-8f25-e5091de1de15" path="/var/lib/kubelet/pods/d5d3b6a8-40d4-404d-8f25-e5091de1de15/volumes" Oct 30 19:42:53.107956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:53.107919 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:54.903057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:54.903017 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.700619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:55.700588 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:55.713105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:55.713062 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:56.242481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:56.242400 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:42:56.524285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:56.524168 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.810757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:57.810717 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.008603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:58.008568 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.169776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:58.169691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.326124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:58.325671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.952249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:58.952205 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.096727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.096690 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 19:42:59.097490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097465 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" Oct 30 19:42:59.097490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097487 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" Oct 30 19:42:59.097734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097521 4990 cpu_manager.go:414] "containerMap: removing container" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097529 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097562 4990 cpu_manager.go:414] "containerMap: removing container" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097570 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097748 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097769 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dec97f-23a4-47bd-980c-11fab281d90c" containerName="istio-proxy" Oct 30 19:42:59.097936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.097791 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3522964f-5fe5-432b-a4ab-7b8e8756d435" containerName="payload-pre-processing" Oct 30 19:42:59.103476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.103453 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.115907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.115869 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6b7ccc-openshift-default-dockercfg-8b86q\"" Oct 30 19:42:59.131606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.131583 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 19:42:59.294141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294047 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294168 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294199 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294328 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294354 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294511 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294652 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294702 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.294788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.294768 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9rq\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.395692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395647 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.395895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395722 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9rq\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.395895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395798 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.395895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395838 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.395895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395867 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.396172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.395970 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.396172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.396007 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.396172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.396122 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.396343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.396224 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.397477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.397446 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.398848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.398734 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.399652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.399628 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.401162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.400546 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.401382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.401362 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.434110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.420518 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.434110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.421977 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.434110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.429211 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9rq\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.460113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.457663 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:42:59.591642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.591604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.720979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:42:59.720944 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:43:00.049803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.049770 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.071306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.071270 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 19:43:00.078966 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:00.078911 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70937de3_8424_4604_8652_ff61aad4a9b4.slice/crio-8afd50a91bb5eb5070b9af6ad38b1c4cd1b46796a9b9ab0f5a6855f363260105 WatchSource:0}: Error finding container 8afd50a91bb5eb5070b9af6ad38b1c4cd1b46796a9b9ab0f5a6855f363260105: Status 404 returned error can't find the container with id 8afd50a91bb5eb5070b9af6ad38b1c4cd1b46796a9b9ab0f5a6855f363260105 Oct 30 19:43:00.082778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.082745 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:00.082888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.082821 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:00.082888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.082872 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:00.265771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.265718 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" event={"ID":"70937de3-8424-4604-8652-ff61aad4a9b4","Type":"ContainerStarted","Data":"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800"} Oct 30 19:43:00.265968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.265820 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" event={"ID":"70937de3-8424-4604-8652-ff61aad4a9b4","Type":"ContainerStarted","Data":"8afd50a91bb5eb5070b9af6ad38b1c4cd1b46796a9b9ab0f5a6855f363260105"} Oct 30 19:43:00.307814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.307690 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podStartSLOduration=1.307672707 podStartE2EDuration="1.307672707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:00.303625486 +0000 UTC m=+2288.015150853" watchObservedRunningTime="2026-10-30 19:43:00.307672707 +0000 UTC m=+2288.019198074" Oct 30 19:43:00.661753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.661660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.721351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.721322 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:43:00.798514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:00.798480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.723119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:01.723056 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:01.723621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:01.723152 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:02.452719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:02.452675 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:02.722317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:02.722222 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:02.722317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:02.722302 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:03.504185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:03.504140 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.699677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:03.699643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.721578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:03.721545 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:03.721750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:03.721606 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:04.459818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:04.459759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:04.722109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:04.722005 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:04.722496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:04.722075 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:05.118120 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:43:05.118071 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:43:05.315788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.315745 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:05.432572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.432491 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 19:43:05.437320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.437296 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.487820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.487784 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:05.539119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.532349 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.539119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.532425 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bn2\" (UniqueName: \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.582785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.582743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 19:43:05.678125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.638432 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.678125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.638498 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bn2\" (UniqueName: \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.678125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.671511 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-095f55-dockercfg-j42kb\"" Oct 30 19:43:05.678125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.671595 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-095f55\"" Oct 30 19:43:05.772466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.772375 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 19:43:05.800112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.785582 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:05.800112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.785643 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:05.804701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.782900 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.818420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.818388 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bn2\" (UniqueName: \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\") pod \"payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:05.837590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.837545 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 19:43:05.837750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.837679 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:05.902489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.902442 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:05.902696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:05.902557 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqsw\" (UniqueName: \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.004130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.004077 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.004555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.004530 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqsw\" (UniqueName: \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.005023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.004996 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.023587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.023524 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqsw\" (UniqueName: \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\") pod \"payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.052722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.052694 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:06.099295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.086436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.157276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.157248 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:06.247779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.246332 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.408740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.408595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.433360 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:06.433323 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfeddfa_aaf2_48bf_86c8_44554170b811.slice/crio-07631ee1aa492a8069bfa132b5b23499c20a48c4e463049b3ee20adf674c6c3b WatchSource:0}: Error finding container 07631ee1aa492a8069bfa132b5b23499c20a48c4e463049b3ee20adf674c6c3b: Status 404 returned error can't find the container with id 07631ee1aa492a8069bfa132b5b23499c20a48c4e463049b3ee20adf674c6c3b Oct 30 19:43:06.444843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.444817 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 19:43:06.562321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.562234 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 19:43:06.563598 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:06.563567 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d153d4_e042_4b49_a64a_25cc4e542a7a.slice/crio-b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795 WatchSource:0}: Error finding container b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795: Status 404 returned error can't find the container with id b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795 Oct 30 19:43:06.721872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.721841 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:43:06.722082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.721889 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 19:43:06.767001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:06.766949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:07.356588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.356552 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" event={"ID":"16d153d4-e042-4b49-a64a-25cc4e542a7a","Type":"ContainerStarted","Data":"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b"} Oct 30 19:43:07.357171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.356649 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:07.357171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.356679 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" event={"ID":"16d153d4-e042-4b49-a64a-25cc4e542a7a","Type":"ContainerStarted","Data":"b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795"} Oct 30 19:43:07.358339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.358313 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" event={"ID":"adfeddfa-aaf2-48bf-86c8-44554170b811","Type":"ContainerStarted","Data":"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8"} Oct 30 19:43:07.358436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.358355 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" event={"ID":"adfeddfa-aaf2-48bf-86c8-44554170b811","Type":"ContainerStarted","Data":"07631ee1aa492a8069bfa132b5b23499c20a48c4e463049b3ee20adf674c6c3b"} Oct 30 19:43:07.358436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.358389 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:07.392781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.392733 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" podStartSLOduration=2.3927194800000002 podStartE2EDuration="2.39271948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:07.391204297 +0000 UTC m=+2295.102729665" watchObservedRunningTime="2026-10-30 19:43:07.39271948 +0000 UTC m=+2295.104244845" Oct 30 19:43:07.428039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.427994 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" podStartSLOduration=2.427979924 podStartE2EDuration="2.427979924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:07.425485841 +0000 UTC m=+2295.137011206" watchObservedRunningTime="2026-10-30 19:43:07.427979924 +0000 UTC m=+2295.139505290" Oct 30 19:43:07.726139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.726053 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:43:07.726518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.726490 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:43:07.727195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:07.727174 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 19:43:08.217065 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:08.217017 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:09.792270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:09.791390 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.556208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:11.554134 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:13.865829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:13.865791 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:14.372913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:14.372859 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:15.671245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:15.671207 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.431927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:17.431855 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:18.369528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:18.369496 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 19:43:18.369713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:18.369582 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 19:43:19.214693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:19.214651 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:20.079197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.079160 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:43:20.084672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.084648 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.105160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.105135 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-08bcfe-openshift-default-dockercfg-hdsw9\"" Oct 30 19:43:20.120595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.120566 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:43:20.143587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143551 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143646 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsk5h\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143700 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143779 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143848 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.143937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.143871 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.144143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.144102 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.144262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.144244 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.245968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.245926 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.245980 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsk5h\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246169 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246201 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246247 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246283 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246388 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246501 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246526 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246602 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246629 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.246854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.246830 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.247037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.247017 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.248425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.248399 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.248535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.248501 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.263182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.263149 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.263408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.263388 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsk5h\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.402678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.402583 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:20.735837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.735809 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:43:20.737343 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:20.737317 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63af8504_30b6_435e_a04c_5e14cac99c80.slice/crio-b151df0fef493d569ed49a3ad6d4e52e74121bc1f238d78709d6c3ca0ff08f14 WatchSource:0}: Error finding container b151df0fef493d569ed49a3ad6d4e52e74121bc1f238d78709d6c3ca0ff08f14: Status 404 returned error can't find the container with id b151df0fef493d569ed49a3ad6d4e52e74121bc1f238d78709d6c3ca0ff08f14 Oct 30 19:43:20.740530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.740497 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:20.740678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.740565 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:20.740678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.740594 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:20.819575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:20.819545 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:21.520484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:21.520451 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" event={"ID":"63af8504-30b6-435e-a04c-5e14cac99c80","Type":"ContainerStarted","Data":"62ccdc5abb2d3740ccb1a98384f8ae50bc249c6cfd00862c19086d11599e9135"} Oct 30 19:43:21.520939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:21.520502 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" event={"ID":"63af8504-30b6-435e-a04c-5e14cac99c80","Type":"ContainerStarted","Data":"b151df0fef493d569ed49a3ad6d4e52e74121bc1f238d78709d6c3ca0ff08f14"} Oct 30 19:43:21.573350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:21.573278 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podStartSLOduration=1.573264515 podStartE2EDuration="1.573264515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:21.568297438 +0000 UTC m=+2309.279822806" watchObservedRunningTime="2026-10-30 19:43:21.573264515 +0000 UTC m=+2309.284789883" Oct 30 19:43:21.996183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:21.996146 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:43:22.002285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.002253 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.016191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.016150 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:43:22.018005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.017981 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5d92fa-openshift-default-dockercfg-9z4jq\"" Oct 30 19:43:22.101760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.101725 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.101990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.101773 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjsr\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.101990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.101798 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.101990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.101943 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.102194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.101999 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.102194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.102127 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.102194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.102156 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.102297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.102220 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.102297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.102245 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205559 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205640 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjsr\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205676 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205749 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205800 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205859 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.205902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205896 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.206190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205965 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.206190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.205997 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.212531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.212495 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.212796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.212773 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.213809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.213780 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.214708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.214681 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.214804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.214757 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.215918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.215872 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.221293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.221271 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.230165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.230141 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.230387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.230365 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjsr\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.324768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.324724 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:22.409172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.406616 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:22.409172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.406721 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:22.419500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.419470 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:22.533440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.533405 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:22.701947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.701920 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:43:22.704993 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:22.704955 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158ecac9_d0f1_4aa7_b714_04b49ea96832.slice/crio-721cdceccde0c561097d70a6cc04ca74a8abbc415a8a3fbfb410c059bb339fe0 WatchSource:0}: Error finding container 721cdceccde0c561097d70a6cc04ca74a8abbc415a8a3fbfb410c059bb339fe0: Status 404 returned error can't find the container with id 721cdceccde0c561097d70a6cc04ca74a8abbc415a8a3fbfb410c059bb339fe0 Oct 30 19:43:22.708030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.707987 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:22.708151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.708072 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:22.708151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:22.708136 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:43:23.403222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:23.403183 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:23.403402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:23.403247 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:23.555662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:23.555625 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" event={"ID":"158ecac9-d0f1-4aa7-b714-04b49ea96832","Type":"ContainerStarted","Data":"f439e4e1a9026365d2148ab72afdb889df3ebf26d3c171d6eef296dbc78538cf"} Oct 30 19:43:23.556157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:23.555688 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" event={"ID":"158ecac9-d0f1-4aa7-b714-04b49ea96832","Type":"ContainerStarted","Data":"721cdceccde0c561097d70a6cc04ca74a8abbc415a8a3fbfb410c059bb339fe0"} Oct 30 19:43:23.624704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:23.624650 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podStartSLOduration=2.624635691 podStartE2EDuration="2.624635691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:23.62383111 +0000 UTC m=+2311.335356477" watchObservedRunningTime="2026-10-30 19:43:23.624635691 +0000 UTC m=+2311.336161056" Oct 30 19:43:24.287119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.283748 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:24.334114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.333466 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:24.334114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.333609 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:24.334114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.333650 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:24.407117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.402964 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:24.407117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:24.403021 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:25.326115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:25.325498 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:25.326115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:25.325577 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:25.343175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:25.339732 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:25.405702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:25.405658 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:25.406027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:25.405981 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:26.328115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:26.327579 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:26.328115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:26.327653 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:26.354229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:26.354145 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:26.411120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:26.407508 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:26.411120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:26.407581 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:27.325458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:27.325411 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:27.325732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:27.325698 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:27.405246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:27.405210 4990 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:43:27.405993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:27.405871 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 19:43:28.145602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.145452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:28.326056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.326018 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:28.326274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.326084 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:28.410780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.410696 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:28.411272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.411245 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:28.411980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:28.411958 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:43:29.325257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:29.325216 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 19:43:29.325440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:29.325288 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 19:43:29.638168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:29.637215 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.150876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:30.150839 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.330643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:30.330606 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:30.331104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:30.331071 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:30.331726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:30.331704 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:43:31.841679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:31.841643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:33.528645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:33.528608 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:35.098605 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:43:35.098568 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:43:35.229533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:35.229384 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:36.977840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:36.977803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:38.668050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:38.668014 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:38.877672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:38.877638 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:39.911664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:39.911625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:40.665689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:40.665640 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.407029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:42.406990 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.133259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:44.133230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.300547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:44.300500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.451509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:44.451425 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:46.160127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:46.157368 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:48.002139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:48.001441 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:49.668890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:49.668842 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.118587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:50.118551 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.608049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:50.607997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:50.801324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:50.800635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.325282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.325250 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:43:51.329781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.329723 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.345051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.345021 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f44f74\"/\"kube-root-ca.crt\"" Oct 30 19:43:51.345203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.345050 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f44f74\"/\"default-dockercfg-ngckh\"" Oct 30 19:43:51.355168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.355142 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f44f74\"/\"auth-test-model-f44f74-kserve-self-signed-certs\"" Oct 30 19:43:51.355318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.355267 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f44f74\"/\"openshift-service-ca.crt\"" Oct 30 19:43:51.361264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.361240 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:43:51.372764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.372731 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwr99\" (UniqueName: \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.372890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.372811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.372957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.372926 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.373011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.372987 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.373079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.373042 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.373079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.373065 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.474877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.474840 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475056 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475200 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475348 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475356 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475425 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475497 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwr99\" (UniqueName: \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.475889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.475865 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.476104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.476067 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.478059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.478038 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.478214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.478196 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.483698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.483656 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.493427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.493403 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwr99\" (UniqueName: \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\") pod \"auth-test-model-f44f74-kserve-d9dd74965-q7482\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.649210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.644743 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:51.684359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:51.684326 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:52.017906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:52.017863 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:43:52.018342 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:52.018220 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod417cd409_7aa8_4d53_a675_adb6a271e717.slice/crio-d19005fa1578ec45cc274f6f8df73efcf8f3941b24426ae452b0c096135b3c91 WatchSource:0}: Error finding container d19005fa1578ec45cc274f6f8df73efcf8f3941b24426ae452b0c096135b3c91: Status 404 returned error can't find the container with id d19005fa1578ec45cc274f6f8df73efcf8f3941b24426ae452b0c096135b3c91 Oct 30 19:43:52.908745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:52.908701 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" event={"ID":"417cd409-7aa8-4d53-a675-adb6a271e717","Type":"ContainerStarted","Data":"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae"} Oct 30 19:43:52.908963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:52.908761 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" event={"ID":"417cd409-7aa8-4d53-a675-adb6a271e717","Type":"ContainerStarted","Data":"d19005fa1578ec45cc274f6f8df73efcf8f3941b24426ae452b0c096135b3c91"} Oct 30 19:43:52.948577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:52.948516 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" podStartSLOduration=1.948498209 podStartE2EDuration="1.948498209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:52.945337841 +0000 UTC m=+2340.656863211" watchObservedRunningTime="2026-10-30 19:43:52.948498209 +0000 UTC m=+2340.660023576" Oct 30 19:43:53.658162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:53.655181 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:43:54.148691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:54.136039 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.600121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:54.600051 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.802194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:54.799798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:55.024052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:55.023949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:56.674528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:56.674488 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.286058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.286022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.379436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.379399 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:43:57.384846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.384828 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.409102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.409058 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:43:57.409493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.409467 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.409660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.409637 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g57v\" (UniqueName: \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.413122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.413084 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-576a62\"" Oct 30 19:43:57.501603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.501454 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.512732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.512697 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g57v\" (UniqueName: \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.514854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.514829 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.516258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.515914 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.566607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.563897 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g57v\" (UniqueName: \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\") pod \"payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:57.700581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:57.700462 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:58.039979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:58.039948 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:43:58.041186 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:43:58.041158 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3e3980_81ef_40bc_9be0_ef75634ffa75.slice/crio-b55c8b23d8079e21669e777b6b44aec2f3b6d6e3d85e7575c0a369eb0992bda9 WatchSource:0}: Error finding container b55c8b23d8079e21669e777b6b44aec2f3b6d6e3d85e7575c0a369eb0992bda9: Status 404 returned error can't find the container with id b55c8b23d8079e21669e777b6b44aec2f3b6d6e3d85e7575c0a369eb0992bda9 Oct 30 19:43:58.831531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:58.831499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.983158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:58.983075 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" event={"ID":"5e3e3980-81ef-40bc-9be0-ef75634ffa75","Type":"ContainerStarted","Data":"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278"} Oct 30 19:43:58.983330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:58.983180 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" event={"ID":"5e3e3980-81ef-40bc-9be0-ef75634ffa75","Type":"ContainerStarted","Data":"b55c8b23d8079e21669e777b6b44aec2f3b6d6e3d85e7575c0a369eb0992bda9"} Oct 30 19:43:58.983330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:58.983210 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:43:59.032500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:43:59.032444 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" podStartSLOduration=2.032425873 podStartE2EDuration="2.032425873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:59.02840704 +0000 UTC m=+2346.739932407" watchObservedRunningTime="2026-10-30 19:43:59.032425873 +0000 UTC m=+2346.743951238" Oct 30 19:44:00.644463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:00.642408 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:00.825482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:00.825443 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:01.645768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:01.645724 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:44:01.659454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:01.659425 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:44:02.028073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:02.027984 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:44:02.401277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:02.401238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:04.073377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:04.073294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.097698 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:44:05.097651 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:44:05.962194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:05.962156 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.678777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:07.678735 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:09.437432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:09.437392 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:09.994666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:09.994631 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:44:11.213281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.213244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:11.632672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.632636 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 19:44:11.637589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.637571 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.672885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.672849 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.673188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.673165 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.673466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.673448 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.673639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.673624 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.673760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.673744 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmdw\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.673899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.673882 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.674058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.674041 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.674295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.674275 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.674488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.674472 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.677234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.677202 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 19:44:11.775822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.775782 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.775903 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.775995 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776027 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmdw\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776086 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776133 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776281 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776372 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776425 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776496 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776570 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776654 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.776869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.776843 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.777328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.777302 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.778653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.778592 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.778773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.778653 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.780299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.780270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:11.793739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.793691 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmdw\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.793856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.793829 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:11.958982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:11.958888 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:12.302879 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:44:12.302835 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dfc325_47ac_4183_9276_b398c5517c54.slice/crio-c41ee14dc2f0273d4c23b81ed78ed11ee31cc583bbe86635579f3c5da8370733 WatchSource:0}: Error finding container c41ee14dc2f0273d4c23b81ed78ed11ee31cc583bbe86635579f3c5da8370733: Status 404 returned error can't find the container with id c41ee14dc2f0273d4c23b81ed78ed11ee31cc583bbe86635579f3c5da8370733 Oct 30 19:44:12.303511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:12.303432 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 19:44:12.306582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:12.306538 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:44:12.306670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:12.306632 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:44:12.306711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:12.306673 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:44:13.027119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.026388 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:13.144878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.144837 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" event={"ID":"d4dfc325-47ac-4183-9276-b398c5517c54","Type":"ContainerStarted","Data":"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02"} Oct 30 19:44:13.145126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.144903 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" event={"ID":"d4dfc325-47ac-4183-9276-b398c5517c54","Type":"ContainerStarted","Data":"c41ee14dc2f0273d4c23b81ed78ed11ee31cc583bbe86635579f3c5da8370733"} Oct 30 19:44:13.208661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.208587 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podStartSLOduration=2.208566909 podStartE2EDuration="2.208566909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:13.196637979 +0000 UTC m=+2360.908163345" watchObservedRunningTime="2026-10-30 19:44:13.208566909 +0000 UTC m=+2360.920092278" Oct 30 19:44:13.960001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.959967 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:13.961806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.961772 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:13.961957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:13.961826 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:14.854361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:14.854294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:14.959330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:14.959294 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:14.959508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:14.959352 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:15.960012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:15.959978 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:15.960427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:15.960027 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:16.764241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:16.764191 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.959854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:16.959817 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:16.960024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:16.959894 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:17.959873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:17.959830 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:17.960422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:17.959898 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:18.564084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:18.564041 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:18.959339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:18.959249 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:44:18.959339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:18.959300 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 19:44:19.963804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:19.963770 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:19.964326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:19.964307 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:19.964983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:19.964960 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 19:44:20.278263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:20.277288 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:22.361296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:22.360929 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:24.450726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:24.450689 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:26.347210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:26.347175 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.364217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:28.364175 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:28.894535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:28.894491 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:44:28.894747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:28.894710 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" containerID="cri-o://2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15" gracePeriod=30 Oct 30 19:44:29.386689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:29.386653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.437766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:30.437729 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:32.286948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:32.286909 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:33.644460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:33.644419 4990 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:44:33.644957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:33.644489 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:44:34.001268 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:44:34.001229 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63745ae5_9ae5_4fcb_9b2b_74d6d19ac0a6.slice/crio-2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:44:34.203433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.203407 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:44:34.240040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240009 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240059 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240112 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240152 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240186 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240265 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240398 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240480 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240472 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240548 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.240668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240572 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\") pod \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\" (UID: \"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6\") " Oct 30 19:44:34.240668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:34.240838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240765 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.240838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.240780 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.241368 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.241343 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.241489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.241370 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.241489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.241384 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.241489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.241398 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.241684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.241662 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.242861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.242835 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "kube-api-access-tqs4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:34.243165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.243139 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:34.243280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.243216 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:34.243545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.243524 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo" pod "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" (UID: "63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:34.263762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.263688 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.343799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.343761 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.343799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.343799 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.344029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.343812 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.344029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.343826 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqs4r\" (UniqueName: \"kubernetes.io/projected/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-kube-api-access-tqs4r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.344029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.343841 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:44:34.383961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.383498 4990 generic.go:356] "Generic (PLEG): container finished" podID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerID="2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15" exitCode=0 Oct 30 19:44:34.383961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.383867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" event={"ID":"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6","Type":"ContainerDied","Data":"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15"} Oct 30 19:44:34.384259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.383991 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" event={"ID":"63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6","Type":"ContainerDied","Data":"da0cbee09e15a801ec30c34c931cd8133fef8bf1dbbee5c84636d4289e5b5378"} Oct 30 19:44:34.384259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.384010 4990 scope.go:122] "RemoveContainer" containerID="2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15" Oct 30 19:44:34.384475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.384452 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws" Oct 30 19:44:34.404241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.404223 4990 scope.go:122] "RemoveContainer" containerID="2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15" Oct 30 19:44:34.404560 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:44:34.404533 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15\": container with ID starting with 2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15 not found: ID does not exist" containerID="2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15" Oct 30 19:44:34.404659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.404573 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15"} err="failed to get container status \"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15\": rpc error: code = NotFound desc = could not find container \"2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15\": container with ID starting with 2fbfb98b2a9c03a0bb39df95e5af4cfac2e144f0796a5c418147f67455954e15 not found: ID does not exist" Oct 30 19:44:34.447341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.447285 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:44:34.456286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:34.456265 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-w8wws"] Oct 30 19:44:35.101859 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:44:35.101822 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:44:35.604446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:35.604407 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.323741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:36.323706 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.407784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:36.407751 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" path="/var/lib/kubelet/pods/63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6/volumes" Oct 30 19:44:38.260027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:38.259988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.068823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:40.068779 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:41.712331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:41.712296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:42.433605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:42.433569 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.206446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:44.206404 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.548339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:44.548007 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.153199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:46.153164 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.299891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:46.299851 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.700776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:46.700733 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.878781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:46.878478 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.312022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:48.311979 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.238135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:50.238055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.596936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:50.596897 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.753198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:50.753155 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.206151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.206114 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.399796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.399764 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.516438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.516363 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 19:44:52.517154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.517138 4990 cpu_manager.go:414] "containerMap: removing container" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" Oct 30 19:44:52.517212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.517156 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" Oct 30 19:44:52.517328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.517319 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="63745ae5-9ae5-4fcb-9b2b-74d6d19ac0a6" containerName="istio-proxy" Oct 30 19:44:52.521125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.521104 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.534229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.534201 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 19:44:52.538660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.538641 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.545201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.545167 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 19:44:52.548674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.548633 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-870193\"" Oct 30 19:44:52.548810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.548752 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-870193-dockercfg-lg4n6\"" Oct 30 19:44:52.569689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.569657 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.569967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.569949 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ks7\" (UniqueName: \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.570258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.570240 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrvf\" (UniqueName: \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.570413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.570399 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.570912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.570898 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 19:44:52.661208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.661174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:52.720669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.720628 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.721109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.721066 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ks7\" (UniqueName: \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.721402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.721385 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrvf\" (UniqueName: \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.721549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.721534 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.733142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.722956 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.748114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.745643 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.789198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.776134 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrvf\" (UniqueName: \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\") pod \"payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:52.790133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.790085 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ks7\" (UniqueName: \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\") pod \"payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.837041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.836731 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:52.872353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:52.872316 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:53.164932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.164898 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 19:44:53.167056 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:44:53.167027 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5485d987_e7d8_4ea8_9d66_0cd034a1c538.slice/crio-aed9dd397597fb2def317e59510e62b3322534e2e36e359d0ba358ab16f806d9 WatchSource:0}: Error finding container aed9dd397597fb2def317e59510e62b3322534e2e36e359d0ba358ab16f806d9: Status 404 returned error can't find the container with id aed9dd397597fb2def317e59510e62b3322534e2e36e359d0ba358ab16f806d9 Oct 30 19:44:53.227759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.227120 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 19:44:53.241617 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:44:53.241580 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe0109d_53ae_43d2_856d_8ded2e81f4cf.slice/crio-3e7fc2368801f65b86e7af583a0bb8439aa1006333b9b8f2194d3d5711ed969b WatchSource:0}: Error finding container 3e7fc2368801f65b86e7af583a0bb8439aa1006333b9b8f2194d3d5711ed969b: Status 404 returned error can't find the container with id 3e7fc2368801f65b86e7af583a0bb8439aa1006333b9b8f2194d3d5711ed969b Oct 30 19:44:53.579284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.579244 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" event={"ID":"dbe0109d-53ae-43d2-856d-8ded2e81f4cf","Type":"ContainerStarted","Data":"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59"} Oct 30 19:44:53.579592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.579308 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" event={"ID":"dbe0109d-53ae-43d2-856d-8ded2e81f4cf","Type":"ContainerStarted","Data":"3e7fc2368801f65b86e7af583a0bb8439aa1006333b9b8f2194d3d5711ed969b"} Oct 30 19:44:53.579592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.579349 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:44:53.581210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.581182 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" event={"ID":"5485d987-e7d8-4ea8-9d66-0cd034a1c538","Type":"ContainerStarted","Data":"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434"} Oct 30 19:44:53.581336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.581230 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:44:53.581336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.581246 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" event={"ID":"5485d987-e7d8-4ea8-9d66-0cd034a1c538","Type":"ContainerStarted","Data":"aed9dd397597fb2def317e59510e62b3322534e2e36e359d0ba358ab16f806d9"} Oct 30 19:44:53.646645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.646573 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" podStartSLOduration=1.6465521330000001 podStartE2EDuration="1.646552133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:53.64347343 +0000 UTC m=+2401.354998854" watchObservedRunningTime="2026-10-30 19:44:53.646552133 +0000 UTC m=+2401.358077499" Oct 30 19:44:53.694736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:53.694641 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" podStartSLOduration=1.69462613 podStartE2EDuration="1.69462613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:53.689871329 +0000 UTC m=+2401.401396694" watchObservedRunningTime="2026-10-30 19:44:53.69462613 +0000 UTC m=+2401.406151542" Oct 30 19:44:54.273076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.273049 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:44:54.282557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.282536 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:44:54.287301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.287276 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:44:54.296507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.296486 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:44:54.466684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.466635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:54.787396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.787359 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:54.955510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:54.955470 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:55.264525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:55.264489 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:44:55.367351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:55.367310 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:44:55.600716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:55.596983 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:55.769228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:55.769189 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.332273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:56.332225 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:58.069523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:58.069485 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.797115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:44:59.790176 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:00.209187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.209152 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:00.216669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.216639 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:00.216836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.216771 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.222509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.222480 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.254427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.254393 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:00.262380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.259592 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:00.262380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.259710 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.264989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.264956 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" containerID="cri-o://9b520886d9a403b733e3eb80f6d44b6bccccc7e2dd918990f59ea27af63ad0d9" gracePeriod=25 Oct 30 19:45:00.279586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279550 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.279817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279794 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2lv\" (UniqueName: \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.279910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279836 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb44f\" (UniqueName: \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.279910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279882 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qff\" (UniqueName: \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.280044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279908 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.280044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.279981 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.280176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.266749 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:00.288741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.286171 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:00.288741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.286210 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:00.288741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.286309 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.299431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.296920 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.299431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.296831 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:00.315133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.314318 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:00.315133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.314366 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:00.315133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.314494 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.315790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.315693 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:00.319473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.319436 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:45:00.327559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.327545 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.338604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.338576 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:00.386053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.373997 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" containerID="cri-o://d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30" gracePeriod=25 Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386584 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386642 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpmp\" (UniqueName: \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386793 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386861 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2lv\" (UniqueName: \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386927 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb44f\" (UniqueName: \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.386978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qff\" (UniqueName: \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387014 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387141 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387194 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387316 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387412 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4slr\" (UniqueName: \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387498 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9cz\" (UniqueName: \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387547 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kr9\" (UniqueName: \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.388144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.387605 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.390641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.388599 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.390641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.389564 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.390641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.390237 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.391969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.391938 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:00.392195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.392071 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.417122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.412265 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.418582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.418556 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb44f\" (UniqueName: \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.418961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.418930 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qff\" (UniqueName: \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.421336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.420804 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2lv\" (UniqueName: \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\") pod \"maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.436660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.436636 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:00.436854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.436840 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:00.436971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.436961 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:00.438280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.438260 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:00.445546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.445460 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:00.452234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.452210 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:00.458855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.458832 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.461832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.460393 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:00.465195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.465175 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.482281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.482258 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:00.486160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.486142 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:00.489809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.489786 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9brm\" (UniqueName: \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.489910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.489816 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dfl\" (UniqueName: \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.490029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490011 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.490113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490067 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxb9t\" (UniqueName: \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.490275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490256 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.490366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490339 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4slr\" (UniqueName: \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.490430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490418 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.490554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490538 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9cz\" (UniqueName: \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.490624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490577 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kr9\" (UniqueName: \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.490682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490631 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkgx\" (UniqueName: \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.490741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490696 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.490802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490781 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.490847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490765 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.490926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490887 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.490989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490931 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpmp\" (UniqueName: \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.490989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.490963 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.491116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.491076 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.491561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.491536 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.491708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.491689 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.491998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.491976 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.509253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.509227 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9cz\" (UniqueName: \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.513904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.513873 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpmp\" (UniqueName: \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.514293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.514274 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kr9\" (UniqueName: \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.517155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.517081 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4slr\" (UniqueName: \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\") pod \"maas-api-key-cleanup-29889825-5n8gz\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.538594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.538575 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:00.550253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.550234 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:00.589998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.589963 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:00.594363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.594338 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxb9t\" (UniqueName: \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.594594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.594580 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.594713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.594699 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.594837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.594824 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkgx\" (UniqueName: \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.594928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.594917 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.595048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.595037 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.595238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.595224 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9brm\" (UniqueName: \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.595338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.595325 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dfl\" (UniqueName: \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.595502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.595478 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.596325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.596295 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.596593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.596576 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.599135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.597162 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.613814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.613785 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9brm\" (UniqueName: \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.619998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.618292 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dfl\" (UniqueName: \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.620828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.620794 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxb9t\" (UniqueName: \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\") pod \"maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.623777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.623753 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:00.625540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.625509 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkgx\" (UniqueName: \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.637690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.637327 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:00.701133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.701052 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:00.724075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.723847 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:00.753950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.753741 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:00.764186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.763259 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:00.780031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.779681 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:00.783507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.782858 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:00.927239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:00.924313 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.084496 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.084452 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e02b96_75a1_48fc_a40a_e705ef4c8ddc.slice/crio-848bb7c91c3a5a52ff37ddafd9c73c99277393df1afedae0d87bef21e88be827 WatchSource:0}: Error finding container 848bb7c91c3a5a52ff37ddafd9c73c99277393df1afedae0d87bef21e88be827: Status 404 returned error can't find the container with id 848bb7c91c3a5a52ff37ddafd9c73c99277393df1afedae0d87bef21e88be827 Oct 30 19:45:01.085440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.085416 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:01.113541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.112611 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:01.154558 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.154520 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa19767_d08d_49d4_a2bb_d4d1302879e9.slice/crio-c62301289100381e0569b87fe6fbae84c111a1dbec50cb1bcf7b4c6c9803ccce WatchSource:0}: Error finding container c62301289100381e0569b87fe6fbae84c111a1dbec50cb1bcf7b4c6c9803ccce: Status 404 returned error can't find the container with id c62301289100381e0569b87fe6fbae84c111a1dbec50cb1bcf7b4c6c9803ccce Oct 30 19:45:01.160625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.160597 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:01.220438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.217525 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:01.220438 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.219007 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bc570f_670a_4b7c_b71b_ab3a7914a823.slice/crio-63da3663481c1c8fc53fc0342163113173a405897a4f2057414d0c6be5766b4f WatchSource:0}: Error finding container 63da3663481c1c8fc53fc0342163113173a405897a4f2057414d0c6be5766b4f: Status 404 returned error can't find the container with id 63da3663481c1c8fc53fc0342163113173a405897a4f2057414d0c6be5766b4f Oct 30 19:45:01.220438 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.220327 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330ce4c_4834_4b6e_94af_b2370c7273b3.slice/crio-78807b92ed61de5062edbfb65a811da93b4addee7a1a861e609f27e568c29889 WatchSource:0}: Error finding container 78807b92ed61de5062edbfb65a811da93b4addee7a1a861e609f27e568c29889: Status 404 returned error can't find the container with id 78807b92ed61de5062edbfb65a811da93b4addee7a1a861e609f27e568c29889 Oct 30 19:45:01.223036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.223013 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:01.285323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.280546 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.285323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.282574 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:01.404854 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.404818 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729ea970_3ebb_41fa_8c70_26ed3270af88.slice/crio-70a37d9d0d7bd333dce33b8cf5b2697198a5fae04929f63e90b1be723b77758f WatchSource:0}: Error finding container 70a37d9d0d7bd333dce33b8cf5b2697198a5fae04929f63e90b1be723b77758f: Status 404 returned error can't find the container with id 70a37d9d0d7bd333dce33b8cf5b2697198a5fae04929f63e90b1be723b77758f Oct 30 19:45:01.407847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.407818 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:01.417228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.417202 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:01.419859 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.419831 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c7380e_7d52_4b6a_b196_278c1f53bf15.slice/crio-bf614b3bc4e39bcb81b924c97c2564011a3f4695d9b872d8dfba1f16aca88e96 WatchSource:0}: Error finding container bf614b3bc4e39bcb81b924c97c2564011a3f4695d9b872d8dfba1f16aca88e96: Status 404 returned error can't find the container with id bf614b3bc4e39bcb81b924c97c2564011a3f4695d9b872d8dfba1f16aca88e96 Oct 30 19:45:01.439111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.434125 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:01.448562 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.441934 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa571faa_7710_4202_9bb0_2eaf0b1312e9.slice/crio-a99415eda1a04e51b68a2a93adc03e8f01df6e4b5f9da534b10b31d08e4c7f2e WatchSource:0}: Error finding container a99415eda1a04e51b68a2a93adc03e8f01df6e4b5f9da534b10b31d08e4c7f2e: Status 404 returned error can't find the container with id a99415eda1a04e51b68a2a93adc03e8f01df6e4b5f9da534b10b31d08e4c7f2e Oct 30 19:45:01.448562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.447629 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:01.495835 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:45:01.495792 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9cf908_cf47_4206_aa75_fcf4ed2aa937.slice/crio-db0c7675643966c2011b111061ff14df079c4394e2b9882b2ff42f534d3b560d WatchSource:0}: Error finding container db0c7675643966c2011b111061ff14df079c4394e2b9882b2ff42f534d3b560d: Status 404 returned error can't find the container with id db0c7675643966c2011b111061ff14df079c4394e2b9882b2ff42f534d3b560d Oct 30 19:45:01.496828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.496800 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:01.677610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.677577 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerStarted","Data":"3b7586b8dd3c30c867c8409141209dd96227f724bea918c07962f535e1a56f34"} Oct 30 19:45:01.679189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.679148 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerStarted","Data":"db0c7675643966c2011b111061ff14df079c4394e2b9882b2ff42f534d3b560d"} Oct 30 19:45:01.681029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.681002 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerStarted","Data":"63da3663481c1c8fc53fc0342163113173a405897a4f2057414d0c6be5766b4f"} Oct 30 19:45:01.684657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.684630 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerStarted","Data":"70a37d9d0d7bd333dce33b8cf5b2697198a5fae04929f63e90b1be723b77758f"} Oct 30 19:45:01.687023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.686997 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerStarted","Data":"78807b92ed61de5062edbfb65a811da93b4addee7a1a861e609f27e568c29889"} Oct 30 19:45:01.694504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.694141 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerStarted","Data":"bf614b3bc4e39bcb81b924c97c2564011a3f4695d9b872d8dfba1f16aca88e96"} Oct 30 19:45:01.696145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.696069 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerStarted","Data":"848bb7c91c3a5a52ff37ddafd9c73c99277393df1afedae0d87bef21e88be827"} Oct 30 19:45:01.698241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.698154 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerStarted","Data":"203190b4ceccf4b2e0045dbd0cc60ae2567b850a9c8cbe501a16849590003fed"} Oct 30 19:45:01.700626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.700558 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerStarted","Data":"c62301289100381e0569b87fe6fbae84c111a1dbec50cb1bcf7b4c6c9803ccce"} Oct 30 19:45:01.711294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.711196 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerStarted","Data":"a99415eda1a04e51b68a2a93adc03e8f01df6e4b5f9da534b10b31d08e4c7f2e"} Oct 30 19:45:01.724810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.724375 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.734317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:01.733335 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerStarted","Data":"3948b9dc906e14af87ede03e2dca255d1a56b1fd8c7dcf77190bfc2eed5e9ec2"} Oct 30 19:45:02.746851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.746813 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerStarted","Data":"ba003c01e57d4a561e4145256da85bacdaf76562f9f4ab5dfd7195f1af8146f3"} Oct 30 19:45:02.748525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.748501 4990 generic.go:356] "Generic (PLEG): container finished" podID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerID="0ec55a1e2c1ea43491edf409c0611ec792e134db9df98691c146475513bd6b9a" exitCode=60 Oct 30 19:45:02.748649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.748576 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerDied","Data":"0ec55a1e2c1ea43491edf409c0611ec792e134db9df98691c146475513bd6b9a"} Oct 30 19:45:02.748892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.748873 4990 scope.go:122] "RemoveContainer" containerID="0ec55a1e2c1ea43491edf409c0611ec792e134db9df98691c146475513bd6b9a" Oct 30 19:45:02.750503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.750469 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerStarted","Data":"3c0fc55b490eec106e5ca9189058f004ba1fbbeedaa24fa74fc7e3cb6f7bb584"} Oct 30 19:45:02.752330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.752308 4990 generic.go:356] "Generic (PLEG): container finished" podID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerID="57194a06779efbd9221e4418ae6d1d6a8029bd726c02c92938323309d8b51bc3" exitCode=60 Oct 30 19:45:02.752438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.752376 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerDied","Data":"57194a06779efbd9221e4418ae6d1d6a8029bd726c02c92938323309d8b51bc3"} Oct 30 19:45:02.752690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.752608 4990 scope.go:122] "RemoveContainer" containerID="57194a06779efbd9221e4418ae6d1d6a8029bd726c02c92938323309d8b51bc3" Oct 30 19:45:02.754063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.754039 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerStarted","Data":"6e2371e55745babe605f5ae8b049984d34dfcd0794874a94ee4cba452e87f6d6"} Oct 30 19:45:02.755654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.755632 4990 generic.go:356] "Generic (PLEG): container finished" podID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerID="97caf59511fe6329088bf2c93b53a2b492e0a69642c503f3aa835572ac81eff8" exitCode=60 Oct 30 19:45:02.755754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.755707 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerDied","Data":"97caf59511fe6329088bf2c93b53a2b492e0a69642c503f3aa835572ac81eff8"} Oct 30 19:45:02.755908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.755893 4990 scope.go:122] "RemoveContainer" containerID="97caf59511fe6329088bf2c93b53a2b492e0a69642c503f3aa835572ac81eff8" Oct 30 19:45:02.757516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.757494 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerID="ec2205d1f7bdf3332b11ee9a8ee1c60454887532889e7eb0d8d8a04ea5b1db1c" exitCode=60 Oct 30 19:45:02.757615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.757573 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerDied","Data":"ec2205d1f7bdf3332b11ee9a8ee1c60454887532889e7eb0d8d8a04ea5b1db1c"} Oct 30 19:45:02.757764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.757749 4990 scope.go:122] "RemoveContainer" containerID="ec2205d1f7bdf3332b11ee9a8ee1c60454887532889e7eb0d8d8a04ea5b1db1c" Oct 30 19:45:02.759193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.759171 4990 generic.go:356] "Generic (PLEG): container finished" podID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerID="e9d98c632d97b9b2d7ecb0e79c2371cfb0d3524fcfe4b61309be63872a42fd14" exitCode=60 Oct 30 19:45:02.759323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.759267 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerDied","Data":"e9d98c632d97b9b2d7ecb0e79c2371cfb0d3524fcfe4b61309be63872a42fd14"} Oct 30 19:45:02.759486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.759470 4990 scope.go:122] "RemoveContainer" containerID="e9d98c632d97b9b2d7ecb0e79c2371cfb0d3524fcfe4b61309be63872a42fd14" Oct 30 19:45:02.762167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.762142 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerStarted","Data":"efb846e91c320eb6e649f7159f8b7e3f89a2b6e946ae8a390126d964fbf8d449"} Oct 30 19:45:02.769276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.768579 4990 generic.go:356] "Generic (PLEG): container finished" podID="46245223-da25-4bae-89bb-fa85f68514a1" containerID="1fa651a3ca272990a3b58f1ab600167c7383fcb2ca81d625cf9a6a759593ad01" exitCode=60 Oct 30 19:45:02.769276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.768748 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerDied","Data":"1fa651a3ca272990a3b58f1ab600167c7383fcb2ca81d625cf9a6a759593ad01"} Oct 30 19:45:02.772740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.769713 4990 scope.go:122] "RemoveContainer" containerID="1fa651a3ca272990a3b58f1ab600167c7383fcb2ca81d625cf9a6a759593ad01" Oct 30 19:45:02.789081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.788461 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerStarted","Data":"b7ed474115c103f49d21c8ba06509a1bee8c199fd8d5375bf4267162f0eb72d0"} Oct 30 19:45:02.789081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.788687 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" podStartSLOduration=2.7886699520000002 podStartE2EDuration="2.788669952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.783831302 +0000 UTC m=+2410.495356669" watchObservedRunningTime="2026-10-30 19:45:02.788669952 +0000 UTC m=+2410.500195319" Oct 30 19:45:02.856226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:02.856173 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" podStartSLOduration=2.856156067 podStartE2EDuration="2.856156067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.855556448 +0000 UTC m=+2410.567081814" watchObservedRunningTime="2026-10-30 19:45:02.856156067 +0000 UTC m=+2410.567681433" Oct 30 19:45:03.019930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.019865 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" podStartSLOduration=3.019845258 podStartE2EDuration="3.019845258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.016332691 +0000 UTC m=+2410.727858058" watchObservedRunningTime="2026-10-30 19:45:03.019845258 +0000 UTC m=+2410.731370631" Oct 30 19:45:03.124078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.124013 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" podStartSLOduration=3.123994953 podStartE2EDuration="3.123994953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.117630923 +0000 UTC m=+2410.829156289" watchObservedRunningTime="2026-10-30 19:45:03.123994953 +0000 UTC m=+2410.835520320" Oct 30 19:45:03.150126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.150085 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:45:03.150346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.150320 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" podUID="f0029510-1083-4fec-935d-03d20bcdda8b" containerName="maas-api" containerID="cri-o://35ed134d8b54fc02ade0969c5593db03ce4bddee106471ca2beff30c40e2e6d9" gracePeriod=30 Oct 30 19:45:03.220789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.220689 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" podStartSLOduration=3.220670707 podStartE2EDuration="3.220670707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.17444642 +0000 UTC m=+2410.885971786" watchObservedRunningTime="2026-10-30 19:45:03.220670707 +0000 UTC m=+2410.932196076" Oct 30 19:45:03.277749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.277713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:03.918401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.917345 4990 generic.go:356] "Generic (PLEG): container finished" podID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerID="3c0fc55b490eec106e5ca9189058f004ba1fbbeedaa24fa74fc7e3cb6f7bb584" exitCode=60 Oct 30 19:45:03.918401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.917494 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerDied","Data":"3c0fc55b490eec106e5ca9189058f004ba1fbbeedaa24fa74fc7e3cb6f7bb584"} Oct 30 19:45:03.918401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.918110 4990 scope.go:122] "RemoveContainer" containerID="3c0fc55b490eec106e5ca9189058f004ba1fbbeedaa24fa74fc7e3cb6f7bb584" Oct 30 19:45:03.930496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.930472 4990 generic.go:356] "Generic (PLEG): container finished" podID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerID="6e2371e55745babe605f5ae8b049984d34dfcd0794874a94ee4cba452e87f6d6" exitCode=60 Oct 30 19:45:03.930710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.930689 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerDied","Data":"6e2371e55745babe605f5ae8b049984d34dfcd0794874a94ee4cba452e87f6d6"} Oct 30 19:45:03.933167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.933146 4990 scope.go:122] "RemoveContainer" containerID="6e2371e55745babe605f5ae8b049984d34dfcd0794874a94ee4cba452e87f6d6" Oct 30 19:45:03.936394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.936127 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerStarted","Data":"3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860"} Oct 30 19:45:03.939348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.939319 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerStarted","Data":"78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9"} Oct 30 19:45:03.950305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.950287 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:45:03.952396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.952372 4990 generic.go:356] "Generic (PLEG): container finished" podID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerID="efb846e91c320eb6e649f7159f8b7e3f89a2b6e946ae8a390126d964fbf8d449" exitCode=60 Oct 30 19:45:03.952505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.952450 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerDied","Data":"efb846e91c320eb6e649f7159f8b7e3f89a2b6e946ae8a390126d964fbf8d449"} Oct 30 19:45:03.952685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.952667 4990 scope.go:122] "RemoveContainer" containerID="efb846e91c320eb6e649f7159f8b7e3f89a2b6e946ae8a390126d964fbf8d449" Oct 30 19:45:03.964087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.964063 4990 generic.go:356] "Generic (PLEG): container finished" podID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerID="b7ed474115c103f49d21c8ba06509a1bee8c199fd8d5375bf4267162f0eb72d0" exitCode=60 Oct 30 19:45:03.964268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.964121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerDied","Data":"b7ed474115c103f49d21c8ba06509a1bee8c199fd8d5375bf4267162f0eb72d0"} Oct 30 19:45:03.964405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.964385 4990 scope.go:122] "RemoveContainer" containerID="b7ed474115c103f49d21c8ba06509a1bee8c199fd8d5375bf4267162f0eb72d0" Oct 30 19:45:03.966119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.966082 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerID="ba003c01e57d4a561e4145256da85bacdaf76562f9f4ab5dfd7195f1af8146f3" exitCode=60 Oct 30 19:45:03.966364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.966142 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerDied","Data":"ba003c01e57d4a561e4145256da85bacdaf76562f9f4ab5dfd7195f1af8146f3"} Oct 30 19:45:03.966470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.966441 4990 scope.go:122] "RemoveContainer" containerID="ba003c01e57d4a561e4145256da85bacdaf76562f9f4ab5dfd7195f1af8146f3" Oct 30 19:45:03.969127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.969079 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerStarted","Data":"1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d"} Oct 30 19:45:03.971428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.971061 4990 generic.go:356] "Generic (PLEG): container finished" podID="f0029510-1083-4fec-935d-03d20bcdda8b" containerID="35ed134d8b54fc02ade0969c5593db03ce4bddee106471ca2beff30c40e2e6d9" exitCode=0 Oct 30 19:45:03.971428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.971106 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" event={"ID":"f0029510-1083-4fec-935d-03d20bcdda8b","Type":"ContainerDied","Data":"35ed134d8b54fc02ade0969c5593db03ce4bddee106471ca2beff30c40e2e6d9"} Oct 30 19:45:03.971428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.971147 4990 scope.go:122] "RemoveContainer" containerID="35ed134d8b54fc02ade0969c5593db03ce4bddee106471ca2beff30c40e2e6d9" Oct 30 19:45:03.971428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:03.971293 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt" Oct 30 19:45:04.006811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.006765 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.028832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.028787 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" podStartSLOduration=4.028773653 podStartE2EDuration="4.028773653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.027301021 +0000 UTC m=+2411.738826387" watchObservedRunningTime="2026-10-30 19:45:04.028773653 +0000 UTC m=+2411.740299019" Oct 30 19:45:04.039441 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.039419 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\") pod \"f0029510-1083-4fec-935d-03d20bcdda8b\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " Oct 30 19:45:04.039562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.039546 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\" (UniqueName: \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\") pod \"f0029510-1083-4fec-935d-03d20bcdda8b\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " Oct 30 19:45:04.039658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.039639 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\") pod \"f0029510-1083-4fec-935d-03d20bcdda8b\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " Oct 30 19:45:04.039765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.039743 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\") pod \"f0029510-1083-4fec-935d-03d20bcdda8b\" (UID: \"f0029510-1083-4fec-935d-03d20bcdda8b\") " Oct 30 19:45:04.040085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.040064 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca" pod "f0029510-1083-4fec-935d-03d20bcdda8b" (UID: "f0029510-1083-4fec-935d-03d20bcdda8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:04.041348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.040759 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0029510-1083-4fec-935d-03d20bcdda8b-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:04.041608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.041590 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls" pod "f0029510-1083-4fec-935d-03d20bcdda8b" (UID: "f0029510-1083-4fec-935d-03d20bcdda8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:04.041893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.041872 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs" pod "f0029510-1083-4fec-935d-03d20bcdda8b" (UID: "f0029510-1083-4fec-935d-03d20bcdda8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:04.041933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.041888 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm" pod "f0029510-1083-4fec-935d-03d20bcdda8b" (UID: "f0029510-1083-4fec-935d-03d20bcdda8b"). InnerVolumeSpecName "kube-api-access-czcxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:04.064584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.064544 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" podStartSLOduration=4.064531453 podStartE2EDuration="4.064531453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.060212222 +0000 UTC m=+2411.771737587" watchObservedRunningTime="2026-10-30 19:45:04.064531453 +0000 UTC m=+2411.776056818" Oct 30 19:45:04.099522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.099463 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" podStartSLOduration=4.099447594 podStartE2EDuration="4.099447594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.09549535 +0000 UTC m=+2411.807020718" watchObservedRunningTime="2026-10-30 19:45:04.099447594 +0000 UTC m=+2411.810972960" Oct 30 19:45:04.142238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.142197 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czcxm\" (UniqueName: \"kubernetes.io/projected/f0029510-1083-4fec-935d-03d20bcdda8b-kube-api-access-czcxm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:04.142238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.142231 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:04.142434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.142248 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0029510-1083-4fec-935d-03d20bcdda8b-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:04.164563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.164508 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" podStartSLOduration=4.164491411 podStartE2EDuration="4.164491411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.163430552 +0000 UTC m=+2411.874955919" watchObservedRunningTime="2026-10-30 19:45:04.164491411 +0000 UTC m=+2411.876016778" Oct 30 19:45:04.222671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.222636 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.302469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.302408 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" podStartSLOduration=4.302388967 podStartE2EDuration="4.302388967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.24799912 +0000 UTC m=+2411.959524487" watchObservedRunningTime="2026-10-30 19:45:04.302388967 +0000 UTC m=+2412.013914334" Oct 30 19:45:04.390008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.389817 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.411787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.411727 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" podStartSLOduration=4.411712105 podStartE2EDuration="4.411712105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.408813937 +0000 UTC m=+2412.120339303" watchObservedRunningTime="2026-10-30 19:45:04.411712105 +0000 UTC m=+2412.123237470" Oct 30 19:45:04.594147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.594118 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 19:45:04.594284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.594246 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 19:45:04.744051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.743913 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.788799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.788765 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:45:04.793658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.793630 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5d92fa-54bf7b48f4-v78lt"] Oct 30 19:45:04.982740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.982649 4990 generic.go:356] "Generic (PLEG): container finished" podID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerID="9cf11ebfd36ef74b8e9ad0f06b076018a755814a986ce98e6ab6ff2d03a907a6" exitCode=60 Oct 30 19:45:04.983202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.982732 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerDied","Data":"9cf11ebfd36ef74b8e9ad0f06b076018a755814a986ce98e6ab6ff2d03a907a6"} Oct 30 19:45:04.983202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.982805 4990 scope.go:122] "RemoveContainer" containerID="57194a06779efbd9221e4418ae6d1d6a8029bd726c02c92938323309d8b51bc3" Oct 30 19:45:04.983311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.983229 4990 scope.go:122] "RemoveContainer" containerID="9cf11ebfd36ef74b8e9ad0f06b076018a755814a986ce98e6ab6ff2d03a907a6" Oct 30 19:45:04.983664 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:04.983634 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp_odh-ai-gateway-infra(a330ce4c-4834-4b6e-94af-b2370c7273b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" Oct 30 19:45:04.985758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.985730 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerStarted","Data":"a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280"} Oct 30 19:45:04.988698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.988674 4990 generic.go:356] "Generic (PLEG): container finished" podID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerID="3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860" exitCode=60 Oct 30 19:45:04.988869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.988842 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerDied","Data":"3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860"} Oct 30 19:45:04.989424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.989401 4990 scope.go:122] "RemoveContainer" containerID="3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860" Oct 30 19:45:04.990033 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:04.990006 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x_odh-ai-gateway-infra(48e02b96-75a1-48fc-a40a-e705ef4c8ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" Oct 30 19:45:04.991801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.991777 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerID="78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9" exitCode=60 Oct 30 19:45:04.991925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.991897 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerDied","Data":"78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9"} Oct 30 19:45:04.992266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.992249 4990 scope.go:122] "RemoveContainer" containerID="78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9" Oct 30 19:45:04.992941 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:04.992911 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4_odh-ai-gateway-infra(ef6f7330-efcc-4af9-94f6-c46fdce04047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" Oct 30 19:45:04.996278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.996254 4990 generic.go:356] "Generic (PLEG): container finished" podID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerID="7d8039d30532c1e98d1c65df5ef4da8863578cd103d2f43344f1baa0dfe9ee87" exitCode=60 Oct 30 19:45:04.996380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.996347 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerDied","Data":"7d8039d30532c1e98d1c65df5ef4da8863578cd103d2f43344f1baa0dfe9ee87"} Oct 30 19:45:04.996710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.996692 4990 scope.go:122] "RemoveContainer" containerID="7d8039d30532c1e98d1c65df5ef4da8863578cd103d2f43344f1baa0dfe9ee87" Oct 30 19:45:04.997012 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:04.996989 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm_odh-ai-gateway-infra(8fa19767-d08d-49d4-a2bb-d4d1302879e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" Oct 30 19:45:04.999086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:04.999062 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerStarted","Data":"ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a"} Oct 30 19:45:05.001941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.001923 4990 scope.go:122] "RemoveContainer" containerID="97caf59511fe6329088bf2c93b53a2b492e0a69642c503f3aa835572ac81eff8" Oct 30 19:45:05.002465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.002447 4990 generic.go:356] "Generic (PLEG): container finished" podID="46245223-da25-4bae-89bb-fa85f68514a1" containerID="d032c39f48e9ad4d7892048c6e0560baaa10317c04f529b7fd3f6581a23dcf34" exitCode=60 Oct 30 19:45:05.002567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.002515 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerDied","Data":"d032c39f48e9ad4d7892048c6e0560baaa10317c04f529b7fd3f6581a23dcf34"} Oct 30 19:45:05.002887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.002820 4990 scope.go:122] "RemoveContainer" containerID="d032c39f48e9ad4d7892048c6e0560baaa10317c04f529b7fd3f6581a23dcf34" Oct 30 19:45:05.003167 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:05.003144 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm_odh-ai-gateway-infra(46245223-da25-4bae-89bb-fa85f68514a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" podUID="46245223-da25-4bae-89bb-fa85f68514a1" Oct 30 19:45:05.004930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.004905 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerStarted","Data":"0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf"} Oct 30 19:45:05.007151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.007116 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerStarted","Data":"711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6"} Oct 30 19:45:05.009319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.009296 4990 generic.go:356] "Generic (PLEG): container finished" podID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerID="1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d" exitCode=60 Oct 30 19:45:05.009410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.009377 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerDied","Data":"1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d"} Oct 30 19:45:05.009615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.009596 4990 scope.go:122] "RemoveContainer" containerID="1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d" Oct 30 19:45:05.009895 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:05.009868 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc_odh-ai-gateway-infra(66bc570f-670a-4b7c-b71b-ab3a7914a823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" Oct 30 19:45:05.013066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.013041 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerStarted","Data":"ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda"} Oct 30 19:45:05.018600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.018567 4990 scope.go:122] "RemoveContainer" containerID="ec2205d1f7bdf3332b11ee9a8ee1c60454887532889e7eb0d8d8a04ea5b1db1c" Oct 30 19:45:05.037066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.037044 4990 scope.go:122] "RemoveContainer" containerID="e9d98c632d97b9b2d7ecb0e79c2371cfb0d3524fcfe4b61309be63872a42fd14" Oct 30 19:45:05.052121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.052059 4990 scope.go:122] "RemoveContainer" containerID="1fa651a3ca272990a3b58f1ab600167c7383fcb2ca81d625cf9a6a759593ad01" Oct 30 19:45:05.068873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:05.068677 4990 scope.go:122] "RemoveContainer" containerID="0ec55a1e2c1ea43491edf409c0611ec792e134db9df98691c146475513bd6b9a" Oct 30 19:45:05.105793 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:05.105759 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:45:06.031558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.031521 4990 generic.go:356] "Generic (PLEG): container finished" podID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerID="ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda" exitCode=60 Oct 30 19:45:06.032058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.031599 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerDied","Data":"ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda"} Oct 30 19:45:06.032058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.031711 4990 scope.go:122] "RemoveContainer" containerID="3c0fc55b490eec106e5ca9189058f004ba1fbbeedaa24fa74fc7e3cb6f7bb584" Oct 30 19:45:06.032058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.031880 4990 scope.go:122] "RemoveContainer" containerID="ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda" Oct 30 19:45:06.032270 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.032238 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-5n8gz_odh-ai-gateway-infra(729ea970-3ebb-41fa-8c70-26ed3270af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" Oct 30 19:45:06.038021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.037992 4990 generic.go:356] "Generic (PLEG): container finished" podID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerID="a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280" exitCode=60 Oct 30 19:45:06.038125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.038024 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerDied","Data":"a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280"} Oct 30 19:45:06.038603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.038580 4990 scope.go:122] "RemoveContainer" containerID="a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280" Oct 30 19:45:06.039145 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.039082 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5_odh-ai-gateway-infra(f1c7380e-7d52-4b6a-b196-278c1f53bf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" Oct 30 19:45:06.041871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.041852 4990 scope.go:122] "RemoveContainer" containerID="3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860" Oct 30 19:45:06.042182 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.042163 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x_odh-ai-gateway-infra(48e02b96-75a1-48fc-a40a-e705ef4c8ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" Oct 30 19:45:06.043507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.043491 4990 scope.go:122] "RemoveContainer" containerID="78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9" Oct 30 19:45:06.043767 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.043748 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4_odh-ai-gateway-infra(ef6f7330-efcc-4af9-94f6-c46fdce04047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" Oct 30 19:45:06.046938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.046920 4990 generic.go:356] "Generic (PLEG): container finished" podID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerID="ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a" exitCode=60 Oct 30 19:45:06.047032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.046983 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerDied","Data":"ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a"} Oct 30 19:45:06.047288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.047265 4990 scope.go:122] "RemoveContainer" containerID="ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a" Oct 30 19:45:06.047550 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.047532 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f_odh-ai-gateway-infra(aa571faa-7710-4202-9bb0-2eaf0b1312e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" Oct 30 19:45:06.048888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.048877 4990 scope.go:122] "RemoveContainer" containerID="6e2371e55745babe605f5ae8b049984d34dfcd0794874a94ee4cba452e87f6d6" Oct 30 19:45:06.050448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.050423 4990 generic.go:356] "Generic (PLEG): container finished" podID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerID="0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf" exitCode=60 Oct 30 19:45:06.050546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.050484 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerDied","Data":"0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf"} Oct 30 19:45:06.050734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.050717 4990 scope.go:122] "RemoveContainer" containerID="0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf" Oct 30 19:45:06.051019 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.050997 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9_odh-ai-gateway-infra(826b31fe-33ad-4c84-8750-465f77bb2c25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" Oct 30 19:45:06.052401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.052379 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerID="711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6" exitCode=60 Oct 30 19:45:06.052499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.052475 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerDied","Data":"711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6"} Oct 30 19:45:06.052647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.052633 4990 scope.go:122] "RemoveContainer" containerID="711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6" Oct 30 19:45:06.052893 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.052868 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q_odh-ai-gateway-infra(0e9cf908-cf47-4206-aa75-fcf4ed2aa937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" Oct 30 19:45:06.054446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.054430 4990 scope.go:122] "RemoveContainer" containerID="1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d" Oct 30 19:45:06.054648 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:06.054632 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc_odh-ai-gateway-infra(66bc570f-670a-4b7c-b71b-ab3a7914a823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" Oct 30 19:45:06.064532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.064516 4990 scope.go:122] "RemoveContainer" containerID="efb846e91c320eb6e649f7159f8b7e3f89a2b6e946ae8a390126d964fbf8d449" Oct 30 19:45:06.082765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.082743 4990 scope.go:122] "RemoveContainer" containerID="b7ed474115c103f49d21c8ba06509a1bee8c199fd8d5375bf4267162f0eb72d0" Oct 30 19:45:06.100828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.100804 4990 scope.go:122] "RemoveContainer" containerID="ba003c01e57d4a561e4145256da85bacdaf76562f9f4ab5dfd7195f1af8146f3" Oct 30 19:45:06.320118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.320040 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:06.410822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:06.410786 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0029510-1083-4fec-935d-03d20bcdda8b" path="/var/lib/kubelet/pods/f0029510-1083-4fec-935d-03d20bcdda8b/volumes" Oct 30 19:45:07.065935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.065904 4990 scope.go:122] "RemoveContainer" containerID="a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280" Oct 30 19:45:07.066486 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:07.066337 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5_odh-ai-gateway-infra(f1c7380e-7d52-4b6a-b196-278c1f53bf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" Oct 30 19:45:07.068189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.068169 4990 scope.go:122] "RemoveContainer" containerID="ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a" Oct 30 19:45:07.068521 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:07.068496 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f_odh-ai-gateway-infra(aa571faa-7710-4202-9bb0-2eaf0b1312e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" Oct 30 19:45:07.070258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.070232 4990 scope.go:122] "RemoveContainer" containerID="0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf" Oct 30 19:45:07.070529 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:07.070511 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9_odh-ai-gateway-infra(826b31fe-33ad-4c84-8750-465f77bb2c25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" Oct 30 19:45:07.072309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.072290 4990 scope.go:122] "RemoveContainer" containerID="711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6" Oct 30 19:45:07.072557 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:07.072535 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q_odh-ai-gateway-infra(0e9cf908-cf47-4206-aa75-fcf4ed2aa937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" Oct 30 19:45:07.074189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.074169 4990 scope.go:122] "RemoveContainer" containerID="ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda" Oct 30 19:45:07.074472 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:07.074450 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-5n8gz_odh-ai-gateway-infra(729ea970-3ebb-41fa-8c70-26ed3270af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" Oct 30 19:45:07.992991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:07.989992 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.060121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:09.060063 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:09.060541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:09.060074 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:09.764367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:09.764324 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.659631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:10.659595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.808705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:10.808666 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.938023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:10.937938 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:45:10.938240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:10.938217 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" containerID="cri-o://f439e4e1a9026365d2148ab72afdb889df3ebf26d3c171d6eef296dbc78538cf" gracePeriod=30 Oct 30 19:45:11.016266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:11.016230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:11.622870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:11.622829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:13.459548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:13.457404 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:15.333733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:15.333694 4990 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:15.334178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:15.333760 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:15.375150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:15.372210 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:15.396936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:15.396898 4990 scope.go:122] "RemoveContainer" containerID="78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9" Oct 30 19:45:16.210745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.210657 4990 generic.go:356] "Generic (PLEG): container finished" podID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerID="f439e4e1a9026365d2148ab72afdb889df3ebf26d3c171d6eef296dbc78538cf" exitCode=0 Oct 30 19:45:16.210894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.210740 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" event={"ID":"158ecac9-d0f1-4aa7-b714-04b49ea96832","Type":"ContainerDied","Data":"f439e4e1a9026365d2148ab72afdb889df3ebf26d3c171d6eef296dbc78538cf"} Oct 30 19:45:16.213045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.212974 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerStarted","Data":"8010fffcb15fbaf929da40dea74d3786176b55a0425847aeb99d2f4fdb0ba425"} Oct 30 19:45:16.241432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.241410 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:45:16.381671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381639 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381690 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381748 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381823 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.381994 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382026 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382118 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382314 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\") pod \"158ecac9-d0f1-4aa7-b714-04b49ea96832\" (UID: \"158ecac9-d0f1-4aa7-b714-04b49ea96832\") " Oct 30 19:45:16.382383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382307 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.382551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382385 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:16.382607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.382859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.382835 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.383055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383032 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.383509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383484 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.383607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383515 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.383607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383528 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.383607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383543 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158ecac9-d0f1-4aa7-b714-04b49ea96832-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.383607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.383553 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.384421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.384400 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.384685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.384667 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.384943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.384924 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:16.385144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.385124 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr" pod "158ecac9-d0f1-4aa7-b714-04b49ea96832" (UID: "158ecac9-d0f1-4aa7-b714-04b49ea96832"). InnerVolumeSpecName "kube-api-access-rzjsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.393674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.393655 4990 scope.go:122] "RemoveContainer" containerID="7d8039d30532c1e98d1c65df5ef4da8863578cd103d2f43344f1baa0dfe9ee87" Oct 30 19:45:16.393971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.393951 4990 scope.go:122] "RemoveContainer" containerID="1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d" Oct 30 19:45:16.394202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.394176 4990 scope.go:122] "RemoveContainer" containerID="3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860" Oct 30 19:45:16.394514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.394498 4990 scope.go:122] "RemoveContainer" containerID="711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6" Oct 30 19:45:16.485078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.485048 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzjsr\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-kube-api-access-rzjsr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.485222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.485082 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.485222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.485111 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.485222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:16.485126 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158ecac9-d0f1-4aa7-b714-04b49ea96832-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:17.278797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.278755 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:17.288152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.287883 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:17.290467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.290432 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerStarted","Data":"16a147ef402164d5467388fa73f656906ca5cb480cdcf31b1290485e1ee42fb2"} Oct 30 19:45:17.304923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.304891 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerStarted","Data":"5b59c0be7ebe6a789e68d473ed219e35ecbc446fa525636d6d3a63267176fa1d"} Oct 30 19:45:17.320648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.320252 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" event={"ID":"158ecac9-d0f1-4aa7-b714-04b49ea96832","Type":"ContainerDied","Data":"721cdceccde0c561097d70a6cc04ca74a8abbc415a8a3fbfb410c059bb339fe0"} Oct 30 19:45:17.320648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.320328 4990 scope.go:122] "RemoveContainer" containerID="f439e4e1a9026365d2148ab72afdb889df3ebf26d3c171d6eef296dbc78538cf" Oct 30 19:45:17.320648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.320540 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4" Oct 30 19:45:17.330681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.330633 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerStarted","Data":"0de35fdef768ef6f362174058935e76fd5eb322c64b29c81499e463e8fd82695"} Oct 30 19:45:17.335678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.335648 4990 generic.go:356] "Generic (PLEG): container finished" podID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerID="8010fffcb15fbaf929da40dea74d3786176b55a0425847aeb99d2f4fdb0ba425" exitCode=60 Oct 30 19:45:17.335795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.335751 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerDied","Data":"8010fffcb15fbaf929da40dea74d3786176b55a0425847aeb99d2f4fdb0ba425"} Oct 30 19:45:17.355327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.355277 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerStarted","Data":"6ea90d470f0349740b75784446a777dd8e17c061bf2751568587b67ed10bb7a6"} Oct 30 19:45:17.356876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.356828 4990 scope.go:122] "RemoveContainer" containerID="78d717e40bfd39897af8acd86d324381bd970f31fe65429afef3f2de2d774aa9" Oct 30 19:45:17.393565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.393536 4990 scope.go:122] "RemoveContainer" containerID="d032c39f48e9ad4d7892048c6e0560baaa10317c04f529b7fd3f6581a23dcf34" Oct 30 19:45:17.394518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.394480 4990 scope.go:122] "RemoveContainer" containerID="ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a" Oct 30 19:45:17.464654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.464622 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:45:17.475445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.472255 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-xpzt4"] Oct 30 19:45:17.822766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:17.822740 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:18.019708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.019676 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\" (UniqueName: \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\") pod \"ef6f7330-efcc-4af9-94f6-c46fdce04047\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " Oct 30 19:45:18.020027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.020003 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\") pod \"ef6f7330-efcc-4af9-94f6-c46fdce04047\" (UID: \"ef6f7330-efcc-4af9-94f6-c46fdce04047\") " Oct 30 19:45:18.020367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.020340 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca" pod "ef6f7330-efcc-4af9-94f6-c46fdce04047" (UID: "ef6f7330-efcc-4af9-94f6-c46fdce04047"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.020716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.020698 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6f7330-efcc-4af9-94f6-c46fdce04047-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:18.022000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.021979 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx" pod "ef6f7330-efcc-4af9-94f6-c46fdce04047" (UID: "ef6f7330-efcc-4af9-94f6-c46fdce04047"). InnerVolumeSpecName "kube-api-access-txkgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.122064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.122036 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkgx\" (UniqueName: \"kubernetes.io/projected/ef6f7330-efcc-4af9-94f6-c46fdce04047-kube-api-access-txkgx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:18.344300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.344210 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:18.370177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.370145 4990 generic.go:356] "Generic (PLEG): container finished" podID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerID="0de35fdef768ef6f362174058935e76fd5eb322c64b29c81499e463e8fd82695" exitCode=60 Oct 30 19:45:18.370340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.370225 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerDied","Data":"0de35fdef768ef6f362174058935e76fd5eb322c64b29c81499e463e8fd82695"} Oct 30 19:45:18.370340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.370287 4990 scope.go:122] "RemoveContainer" containerID="3bd60f81d4ae4ac71038001c3c1969f1848f43f3be37c7c6d51988b98d0af860" Oct 30 19:45:18.370744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.370719 4990 scope.go:122] "RemoveContainer" containerID="0de35fdef768ef6f362174058935e76fd5eb322c64b29c81499e463e8fd82695" Oct 30 19:45:18.371226 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:18.371185 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x_odh-ai-gateway-infra(48e02b96-75a1-48fc-a40a-e705ef4c8ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" Oct 30 19:45:18.372160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.372130 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" event={"ID":"ef6f7330-efcc-4af9-94f6-c46fdce04047","Type":"ContainerDied","Data":"203190b4ceccf4b2e0045dbd0cc60ae2567b850a9c8cbe501a16849590003fed"} Oct 30 19:45:18.372302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.372184 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4" Oct 30 19:45:18.374669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.374646 4990 generic.go:356] "Generic (PLEG): container finished" podID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerID="6ea90d470f0349740b75784446a777dd8e17c061bf2751568587b67ed10bb7a6" exitCode=60 Oct 30 19:45:18.374957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.374704 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerDied","Data":"6ea90d470f0349740b75784446a777dd8e17c061bf2751568587b67ed10bb7a6"} Oct 30 19:45:18.375194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.375171 4990 scope.go:122] "RemoveContainer" containerID="6ea90d470f0349740b75784446a777dd8e17c061bf2751568587b67ed10bb7a6" Oct 30 19:45:18.375542 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:18.375516 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm_odh-ai-gateway-infra(8fa19767-d08d-49d4-a2bb-d4d1302879e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" Oct 30 19:45:18.377414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.377390 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerStarted","Data":"709e2773d250be1bf7d2d541971d72b70ebce8eea4c8b394f8d2e4c03798faf1"} Oct 30 19:45:18.380388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.380366 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerStarted","Data":"f07f7b4f13ebd53175f7d1315a6f9da8ed089a3b7ab3c32a2f98f7e356274bd9"} Oct 30 19:45:18.382991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.382971 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerID="16a147ef402164d5467388fa73f656906ca5cb480cdcf31b1290485e1ee42fb2" exitCode=60 Oct 30 19:45:18.383317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.383009 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerDied","Data":"16a147ef402164d5467388fa73f656906ca5cb480cdcf31b1290485e1ee42fb2"} Oct 30 19:45:18.385572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.385544 4990 generic.go:356] "Generic (PLEG): container finished" podID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerID="5b59c0be7ebe6a789e68d473ed219e35ecbc446fa525636d6d3a63267176fa1d" exitCode=60 Oct 30 19:45:18.385961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.385589 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerDied","Data":"5b59c0be7ebe6a789e68d473ed219e35ecbc446fa525636d6d3a63267176fa1d"} Oct 30 19:45:18.386048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.385976 4990 scope.go:122] "RemoveContainer" containerID="5b59c0be7ebe6a789e68d473ed219e35ecbc446fa525636d6d3a63267176fa1d" Oct 30 19:45:18.386377 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:18.386355 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc_odh-ai-gateway-infra(66bc570f-670a-4b7c-b71b-ab3a7914a823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" Oct 30 19:45:18.388179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.388153 4990 scope.go:122] "RemoveContainer" containerID="8010fffcb15fbaf929da40dea74d3786176b55a0425847aeb99d2f4fdb0ba425" Oct 30 19:45:18.394957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.394925 4990 scope.go:122] "RemoveContainer" containerID="a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280" Oct 30 19:45:18.395756 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.395716 4990 scope.go:122] "RemoveContainer" containerID="0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf" Oct 30 19:45:18.396387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.396368 4990 scope.go:122] "RemoveContainer" containerID="9cf11ebfd36ef74b8e9ad0f06b076018a755814a986ce98e6ab6ff2d03a907a6" Oct 30 19:45:18.396860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.396837 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:18.397568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.397494 4990 scope.go:122] "RemoveContainer" containerID="ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda" Oct 30 19:45:18.403228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.403192 4990 scope.go:122] "RemoveContainer" containerID="7d8039d30532c1e98d1c65df5ef4da8863578cd103d2f43344f1baa0dfe9ee87" Oct 30 19:45:18.414159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.414133 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" path="/var/lib/kubelet/pods/158ecac9-d0f1-4aa7-b714-04b49ea96832/volumes" Oct 30 19:45:18.442478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.442454 4990 scope.go:122] "RemoveContainer" containerID="711a7feab307bee755c1384a2312e657a071c96033fb626e3b306cdbb3ba1ad6" Oct 30 19:45:18.515317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.514507 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:18.539936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.539361 4990 scope.go:122] "RemoveContainer" containerID="1ab2b9175e4aacf5babede76ce7a7a94f73ecbd3c3d5c847306668e668cb6c1d" Oct 30 19:45:18.586689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.586585 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:18.698140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.698065 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:18.706208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.706182 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889825-7xlk4"] Oct 30 19:45:18.819740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.819698 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:18.845812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.845766 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\") pod \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " Oct 30 19:45:18.845972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.845920 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\" (UniqueName: \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\") pod \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\" (UID: \"0e9cf908-cf47-4206-aa75-fcf4ed2aa937\") " Oct 30 19:45:18.846448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.846413 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca" pod "0e9cf908-cf47-4206-aa75-fcf4ed2aa937" (UID: "0e9cf908-cf47-4206-aa75-fcf4ed2aa937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.846892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.846862 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:18.848612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.848589 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm" pod "0e9cf908-cf47-4206-aa75-fcf4ed2aa937" (UID: "0e9cf908-cf47-4206-aa75-fcf4ed2aa937"). InnerVolumeSpecName "kube-api-access-w9brm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.948235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:18.948155 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9brm\" (UniqueName: \"kubernetes.io/projected/0e9cf908-cf47-4206-aa75-fcf4ed2aa937-kube-api-access-w9brm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:19.040396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.040274 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:19.063330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.061139 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:19.063550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.063366 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:19.401967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.401925 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerStarted","Data":"fe8963f7352d1e6167152b05addfafa526696ad930843b55573d749de6ef5c3c"} Oct 30 19:45:19.403825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.403797 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" event={"ID":"0e9cf908-cf47-4206-aa75-fcf4ed2aa937","Type":"ContainerDied","Data":"db0c7675643966c2011b111061ff14df079c4394e2b9882b2ff42f534d3b560d"} Oct 30 19:45:19.403980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.403842 4990 scope.go:122] "RemoveContainer" containerID="16a147ef402164d5467388fa73f656906ca5cb480cdcf31b1290485e1ee42fb2" Oct 30 19:45:19.403980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.403902 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q" Oct 30 19:45:19.424146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.420059 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerStarted","Data":"9453327b650ed9e214fe70d363cc041122924fa1a4b0507240aff515378793a3"} Oct 30 19:45:19.426836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.425933 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerStarted","Data":"0ea09696fb3423b74626fed68b1086b4623d8319946d7e4264225d1edda8f2ea"} Oct 30 19:45:19.435884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.434165 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerStarted","Data":"02228bd0e506c510d01d55ec170e49c24d99b154796e22caaf12fee414261f89"} Oct 30 19:45:19.472139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.470397 4990 generic.go:356] "Generic (PLEG): container finished" podID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerID="709e2773d250be1bf7d2d541971d72b70ebce8eea4c8b394f8d2e4c03798faf1" exitCode=60 Oct 30 19:45:19.472139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.470490 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerDied","Data":"709e2773d250be1bf7d2d541971d72b70ebce8eea4c8b394f8d2e4c03798faf1"} Oct 30 19:45:19.472139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.470887 4990 scope.go:122] "RemoveContainer" containerID="ee050775c45777d0bf7a4ee78f13f475a62380dc7fa79cab6b2aab474451ab5a" Oct 30 19:45:19.472139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.471461 4990 scope.go:122] "RemoveContainer" containerID="709e2773d250be1bf7d2d541971d72b70ebce8eea4c8b394f8d2e4c03798faf1" Oct 30 19:45:19.472139 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:19.471854 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f_odh-ai-gateway-infra(aa571faa-7710-4202-9bb0-2eaf0b1312e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" Oct 30 19:45:19.475933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.475840 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:19.477988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.477963 4990 generic.go:356] "Generic (PLEG): container finished" podID="46245223-da25-4bae-89bb-fa85f68514a1" containerID="f07f7b4f13ebd53175f7d1315a6f9da8ed089a3b7ab3c32a2f98f7e356274bd9" exitCode=60 Oct 30 19:45:19.478116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.478074 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerDied","Data":"f07f7b4f13ebd53175f7d1315a6f9da8ed089a3b7ab3c32a2f98f7e356274bd9"} Oct 30 19:45:19.478559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.478540 4990 scope.go:122] "RemoveContainer" containerID="f07f7b4f13ebd53175f7d1315a6f9da8ed089a3b7ab3c32a2f98f7e356274bd9" Oct 30 19:45:19.478931 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:19.478898 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm_odh-ai-gateway-infra(46245223-da25-4bae-89bb-fa85f68514a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" podUID="46245223-da25-4bae-89bb-fa85f68514a1" Oct 30 19:45:19.489188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.489074 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:19.521434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.507964 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889825-s7z9q"] Oct 30 19:45:19.521434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.513884 4990 scope.go:122] "RemoveContainer" containerID="d032c39f48e9ad4d7892048c6e0560baaa10317c04f529b7fd3f6581a23dcf34" Oct 30 19:45:19.628487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.628069 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:19.959472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.959390 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:19.969424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.969387 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:19.971353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.971326 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:19.981168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981147 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\" (UniqueName: \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\") pod \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " Oct 30 19:45:19.981430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981415 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\" (UniqueName: \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\") pod \"66bc570f-670a-4b7c-b71b-ab3a7914a823\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " Oct 30 19:45:19.981504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981440 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\" (UniqueName: \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\") pod \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " Oct 30 19:45:19.981504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981467 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\") pod \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\" (UID: \"8fa19767-d08d-49d4-a2bb-d4d1302879e9\") " Oct 30 19:45:19.981614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981518 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\") pod \"66bc570f-670a-4b7c-b71b-ab3a7914a823\" (UID: \"66bc570f-670a-4b7c-b71b-ab3a7914a823\") " Oct 30 19:45:19.981614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.981587 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\") pod \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\" (UID: \"48e02b96-75a1-48fc-a40a-e705ef4c8ddc\") " Oct 30 19:45:19.982064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.982029 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca" pod "8fa19767-d08d-49d4-a2bb-d4d1302879e9" (UID: "8fa19767-d08d-49d4-a2bb-d4d1302879e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:19.982064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.982046 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca" pod "66bc570f-670a-4b7c-b71b-ab3a7914a823" (UID: "66bc570f-670a-4b7c-b71b-ab3a7914a823"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:19.982340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.982313 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca" pod "48e02b96-75a1-48fc-a40a-e705ef4c8ddc" (UID: "48e02b96-75a1-48fc-a40a-e705ef4c8ddc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:19.982603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.982578 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa19767-d08d-49d4-a2bb-d4d1302879e9-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:19.982741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.982604 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bc570f-670a-4b7c-b71b-ab3a7914a823-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:19.983517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.983494 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f" pod "8fa19767-d08d-49d4-a2bb-d4d1302879e9" (UID: "8fa19767-d08d-49d4-a2bb-d4d1302879e9"). InnerVolumeSpecName "kube-api-access-qb44f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:19.983791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.983765 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv" pod "48e02b96-75a1-48fc-a40a-e705ef4c8ddc" (UID: "48e02b96-75a1-48fc-a40a-e705ef4c8ddc"). InnerVolumeSpecName "kube-api-access-7n2lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:19.984086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:19.984065 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz" pod "66bc570f-670a-4b7c-b71b-ab3a7914a823" (UID: "66bc570f-670a-4b7c-b71b-ab3a7914a823"). InnerVolumeSpecName "kube-api-access-9k9cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.084389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.084352 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.084389 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.084392 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2lv\" (UniqueName: \"kubernetes.io/projected/48e02b96-75a1-48fc-a40a-e705ef4c8ddc-kube-api-access-7n2lv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.084585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.084407 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9cz\" (UniqueName: \"kubernetes.io/projected/66bc570f-670a-4b7c-b71b-ab3a7914a823-kube-api-access-9k9cz\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.084585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.084421 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb44f\" (UniqueName: \"kubernetes.io/projected/8fa19767-d08d-49d4-a2bb-d4d1302879e9-kube-api-access-qb44f\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.406297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.406261 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" path="/var/lib/kubelet/pods/0e9cf908-cf47-4206-aa75-fcf4ed2aa937/volumes" Oct 30 19:45:20.407346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.407320 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" path="/var/lib/kubelet/pods/ef6f7330-efcc-4af9-94f6-c46fdce04047/volumes" Oct 30 19:45:20.464748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.464716 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:20.495045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.495015 4990 generic.go:356] "Generic (PLEG): container finished" podID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerID="02228bd0e506c510d01d55ec170e49c24d99b154796e22caaf12fee414261f89" exitCode=60 Oct 30 19:45:20.495185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.495085 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerDied","Data":"02228bd0e506c510d01d55ec170e49c24d99b154796e22caaf12fee414261f89"} Oct 30 19:45:20.495185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.495172 4990 scope.go:122] "RemoveContainer" containerID="a76fd7ea609bbc8c9c709563971358bea7dff1ede1045878355ff0c1e05d9280" Oct 30 19:45:20.495650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.495629 4990 scope.go:122] "RemoveContainer" containerID="02228bd0e506c510d01d55ec170e49c24d99b154796e22caaf12fee414261f89" Oct 30 19:45:20.495985 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:20.495959 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5_odh-ai-gateway-infra(f1c7380e-7d52-4b6a-b196-278c1f53bf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" Oct 30 19:45:20.497644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.497613 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" event={"ID":"48e02b96-75a1-48fc-a40a-e705ef4c8ddc","Type":"ContainerDied","Data":"848bb7c91c3a5a52ff37ddafd9c73c99277393df1afedae0d87bef21e88be827"} Oct 30 19:45:20.497746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.497719 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x" Oct 30 19:45:20.500385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.500361 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" event={"ID":"8fa19767-d08d-49d4-a2bb-d4d1302879e9","Type":"ContainerDied","Data":"c62301289100381e0569b87fe6fbae84c111a1dbec50cb1bcf7b4c6c9803ccce"} Oct 30 19:45:20.500488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.500450 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm" Oct 30 19:45:20.507517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.507491 4990 generic.go:356] "Generic (PLEG): container finished" podID="b08de27d-a43f-4763-9c03-391b03f1a848" containerID="9b520886d9a403b733e3eb80f6d44b6bccccc7e2dd918990f59ea27af63ad0d9" exitCode=0 Oct 30 19:45:20.507749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.507631 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" event={"ID":"b08de27d-a43f-4763-9c03-391b03f1a848","Type":"ContainerDied","Data":"9b520886d9a403b733e3eb80f6d44b6bccccc7e2dd918990f59ea27af63ad0d9"} Oct 30 19:45:20.507749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.507684 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" event={"ID":"b08de27d-a43f-4763-9c03-391b03f1a848","Type":"ContainerDied","Data":"09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710"} Oct 30 19:45:20.507749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.507710 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09814704ec5e5d140b6328d4824441da51130d1b8aaf794f515ad3bfd1b4b710" Oct 30 19:45:20.509774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.509754 4990 generic.go:356] "Generic (PLEG): container finished" podID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerID="fe8963f7352d1e6167152b05addfafa526696ad930843b55573d749de6ef5c3c" exitCode=60 Oct 30 19:45:20.509876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.509839 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerDied","Data":"fe8963f7352d1e6167152b05addfafa526696ad930843b55573d749de6ef5c3c"} Oct 30 19:45:20.514347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.513740 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" event={"ID":"66bc570f-670a-4b7c-b71b-ab3a7914a823","Type":"ContainerDied","Data":"63da3663481c1c8fc53fc0342163113173a405897a4f2057414d0c6be5766b4f"} Oct 30 19:45:20.514347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.513865 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc" Oct 30 19:45:20.523792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.522675 4990 generic.go:356] "Generic (PLEG): container finished" podID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerID="9453327b650ed9e214fe70d363cc041122924fa1a4b0507240aff515378793a3" exitCode=60 Oct 30 19:45:20.523792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.522754 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerDied","Data":"9453327b650ed9e214fe70d363cc041122924fa1a4b0507240aff515378793a3"} Oct 30 19:45:20.523792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.523195 4990 scope.go:122] "RemoveContainer" containerID="9453327b650ed9e214fe70d363cc041122924fa1a4b0507240aff515378793a3" Oct 30 19:45:20.523792 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:20.523486 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-5n8gz_odh-ai-gateway-infra(729ea970-3ebb-41fa-8c70-26ed3270af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" Oct 30 19:45:20.527128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.526849 4990 generic.go:356] "Generic (PLEG): container finished" podID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerID="0ea09696fb3423b74626fed68b1086b4623d8319946d7e4264225d1edda8f2ea" exitCode=60 Oct 30 19:45:20.527128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.526881 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerDied","Data":"0ea09696fb3423b74626fed68b1086b4623d8319946d7e4264225d1edda8f2ea"} Oct 30 19:45:20.527803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.527450 4990 scope.go:122] "RemoveContainer" containerID="0ea09696fb3423b74626fed68b1086b4623d8319946d7e4264225d1edda8f2ea" Oct 30 19:45:20.527803 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:20.527750 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp_odh-ai-gateway-infra(a330ce4c-4834-4b6e-94af-b2370c7273b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" Oct 30 19:45:20.549408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.549311 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:20.554444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.553573 4990 scope.go:122] "RemoveContainer" containerID="0de35fdef768ef6f362174058935e76fd5eb322c64b29c81499e463e8fd82695" Oct 30 19:45:20.554884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.554783 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:45:20.614553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.614312 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\") pod \"b08de27d-a43f-4763-9c03-391b03f1a848\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " Oct 30 19:45:20.614553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.614403 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\") pod \"b08de27d-a43f-4763-9c03-391b03f1a848\" (UID: \"b08de27d-a43f-4763-9c03-391b03f1a848\") " Oct 30 19:45:20.621591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.616470 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume" pod "b08de27d-a43f-4763-9c03-391b03f1a848" (UID: "b08de27d-a43f-4763-9c03-391b03f1a848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.621591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.618689 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:20.630603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.630572 4990 scope.go:122] "RemoveContainer" containerID="6ea90d470f0349740b75784446a777dd8e17c061bf2751568587b67ed10bb7a6" Oct 30 19:45:20.642134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.640809 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:20.662537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.660878 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z" pod "b08de27d-a43f-4763-9c03-391b03f1a848" (UID: "b08de27d-a43f-4763-9c03-391b03f1a848"). InnerVolumeSpecName "kube-api-access-6df9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.701329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.700927 4990 scope.go:122] "RemoveContainer" containerID="0627cc9f3a7e7717799b7613e06dfe9030d21c0f24c59925b7a730399c19c0bf" Oct 30 19:45:20.717175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.717132 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6df9z\" (UniqueName: \"kubernetes.io/projected/b08de27d-a43f-4763-9c03-391b03f1a848-kube-api-access-6df9z\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.717175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.717176 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08de27d-a43f-4763-9c03-391b03f1a848-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.730537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.730511 4990 scope.go:122] "RemoveContainer" containerID="5b59c0be7ebe6a789e68d473ed219e35ecbc446fa525636d6d3a63267176fa1d" Oct 30 19:45:20.773868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.773843 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:45:20.774012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.773995 4990 scope.go:122] "RemoveContainer" containerID="ec0c57611a46dcd1ec4663d49ebab7304e47d5107c521f8cdb173afb123f2fda" Oct 30 19:45:20.808819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.808257 4990 scope.go:122] "RemoveContainer" containerID="9cf11ebfd36ef74b8e9ad0f06b076018a755814a986ce98e6ab6ff2d03a907a6" Oct 30 19:45:20.818273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.818247 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\") pod \"16a8917f-6024-4417-a379-02623868b1eb\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " Oct 30 19:45:20.818919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.818900 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\" (UniqueName: \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\") pod \"16a8917f-6024-4417-a379-02623868b1eb\" (UID: \"16a8917f-6024-4417-a379-02623868b1eb\") " Oct 30 19:45:20.820714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.818771 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume" pod "16a8917f-6024-4417-a379-02623868b1eb" (UID: "16a8917f-6024-4417-a379-02623868b1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.856070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.855638 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj" pod "16a8917f-6024-4417-a379-02623868b1eb" (UID: "16a8917f-6024-4417-a379-02623868b1eb"). InnerVolumeSpecName "kube-api-access-jhqjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.922055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.922030 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqjj\" (UniqueName: \"kubernetes.io/projected/16a8917f-6024-4417-a379-02623868b1eb-kube-api-access-jhqjj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.922055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.922057 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a8917f-6024-4417-a379-02623868b1eb-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:20.947156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.947121 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:20.957728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.957701 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:20.958307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.958282 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:20.970506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.970470 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889825-nwbdm"] Oct 30 19:45:20.973265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:20.973242 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:21.005718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.005558 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:21.017137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.014405 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:21.019834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.019807 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-614c02-29889825-jc26x"] Oct 30 19:45:21.022630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022578 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\") pod \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " Oct 30 19:45:21.022748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022663 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\" (UniqueName: \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\") pod \"826b31fe-33ad-4c84-8750-465f77bb2c25\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " Oct 30 19:45:21.022748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022699 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\") pod \"826b31fe-33ad-4c84-8750-465f77bb2c25\" (UID: \"826b31fe-33ad-4c84-8750-465f77bb2c25\") " Oct 30 19:45:21.022748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022731 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\" (UniqueName: \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\") pod \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\" (UID: \"aa571faa-7710-4202-9bb0-2eaf0b1312e9\") " Oct 30 19:45:21.022856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022848 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\" (UniqueName: \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\") pod \"46245223-da25-4bae-89bb-fa85f68514a1\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " Oct 30 19:45:21.023317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.022928 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\") pod \"46245223-da25-4bae-89bb-fa85f68514a1\" (UID: \"46245223-da25-4bae-89bb-fa85f68514a1\") " Oct 30 19:45:21.023317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.023279 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca" pod "826b31fe-33ad-4c84-8750-465f77bb2c25" (UID: "826b31fe-33ad-4c84-8750-465f77bb2c25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.023473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.023430 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b31fe-33ad-4c84-8750-465f77bb2c25-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.023644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.023617 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca" pod "46245223-da25-4bae-89bb-fa85f68514a1" (UID: "46245223-da25-4bae-89bb-fa85f68514a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.024630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.024606 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca" pod "aa571faa-7710-4202-9bb0-2eaf0b1312e9" (UID: "aa571faa-7710-4202-9bb0-2eaf0b1312e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.025399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.025375 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl" pod "aa571faa-7710-4202-9bb0-2eaf0b1312e9" (UID: "aa571faa-7710-4202-9bb0-2eaf0b1312e9"). InnerVolumeSpecName "kube-api-access-x4dfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.025484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.025437 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff" pod "46245223-da25-4bae-89bb-fa85f68514a1" (UID: "46245223-da25-4bae-89bb-fa85f68514a1"). InnerVolumeSpecName "kube-api-access-k9qff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.025574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.025560 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9" pod "826b31fe-33ad-4c84-8750-465f77bb2c25" (UID: "826b31fe-33ad-4c84-8750-465f77bb2c25"). InnerVolumeSpecName "kube-api-access-42kr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.076348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.076300 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:21.086365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.086330 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889825-qmkcc"] Oct 30 19:45:21.125325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.125283 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46245223-da25-4bae-89bb-fa85f68514a1-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.125325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.125320 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa571faa-7710-4202-9bb0-2eaf0b1312e9-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.125549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.125335 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42kr9\" (UniqueName: \"kubernetes.io/projected/826b31fe-33ad-4c84-8750-465f77bb2c25-kube-api-access-42kr9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.125549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.125350 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dfl\" (UniqueName: \"kubernetes.io/projected/aa571faa-7710-4202-9bb0-2eaf0b1312e9-kube-api-access-x4dfl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.125549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.125364 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qff\" (UniqueName: \"kubernetes.io/projected/46245223-da25-4bae-89bb-fa85f68514a1-kube-api-access-k9qff\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.550668 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.550637 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" event={"ID":"aa571faa-7710-4202-9bb0-2eaf0b1312e9","Type":"ContainerDied","Data":"a99415eda1a04e51b68a2a93adc03e8f01df6e4b5f9da534b10b31d08e4c7f2e"} Oct 30 19:45:21.551055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.550688 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f" Oct 30 19:45:21.551055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.550705 4990 scope.go:122] "RemoveContainer" containerID="709e2773d250be1bf7d2d541971d72b70ebce8eea4c8b394f8d2e4c03798faf1" Oct 30 19:45:21.552568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.552532 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" event={"ID":"46245223-da25-4bae-89bb-fa85f68514a1","Type":"ContainerDied","Data":"3948b9dc906e14af87ede03e2dca255d1a56b1fd8c7dcf77190bfc2eed5e9ec2"} Oct 30 19:45:21.552674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.552621 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm" Oct 30 19:45:21.563193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.563158 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" event={"ID":"826b31fe-33ad-4c84-8750-465f77bb2c25","Type":"ContainerDied","Data":"3b7586b8dd3c30c867c8409141209dd96227f724bea918c07962f535e1a56f34"} Oct 30 19:45:21.563297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.563181 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9" Oct 30 19:45:21.567134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.566609 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" Oct 30 19:45:21.567134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.566628 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" event={"ID":"16a8917f-6024-4417-a379-02623868b1eb","Type":"ContainerDied","Data":"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30"} Oct 30 19:45:21.567134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.566553 4990 generic.go:356] "Generic (PLEG): container finished" podID="16a8917f-6024-4417-a379-02623868b1eb" containerID="d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30" exitCode=0 Oct 30 19:45:21.567134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.566903 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx" Oct 30 19:45:21.567927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.567890 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn" event={"ID":"16a8917f-6024-4417-a379-02623868b1eb","Type":"ContainerDied","Data":"983539631f5e59465a9eb917c44c7b18e05c20043d938bde56294e64ebc0e69a"} Oct 30 19:45:21.577690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.577661 4990 scope.go:122] "RemoveContainer" containerID="f07f7b4f13ebd53175f7d1315a6f9da8ed089a3b7ab3c32a2f98f7e356274bd9" Oct 30 19:45:21.608169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.608141 4990 scope.go:122] "RemoveContainer" containerID="fe8963f7352d1e6167152b05addfafa526696ad930843b55573d749de6ef5c3c" Oct 30 19:45:21.628253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.628235 4990 scope.go:122] "RemoveContainer" containerID="d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30" Oct 30 19:45:21.642578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.642546 4990 scope.go:122] "RemoveContainer" containerID="d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30" Oct 30 19:45:21.643155 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:21.643131 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30\": container with ID starting with d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30 not found: ID does not exist" containerID="d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30" Oct 30 19:45:21.643325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.643296 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30"} err="failed to get container status \"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30\": rpc error: code = NotFound desc = could not find container \"d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30\": container with ID starting with d9b28961bcf0299f143f7613026c1f6813c98549caa5618aa4998e431a97cd30 not found: ID does not exist" Oct 30 19:45:21.656181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.656153 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:21.660683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.660663 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7b7bc-29889825-t24tm"] Oct 30 19:45:21.745831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.745801 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:21.758339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.758313 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889825-f2p8f"] Oct 30 19:45:21.850145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.850122 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:45:21.853410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.853334 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5d92fa-588589ff7f-ql4xn"] Oct 30 19:45:21.856591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.856151 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:21.858660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.858641 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:21.864592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.864559 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889825-n5rg9"] Oct 30 19:45:21.865884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.865862 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:21.870589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.870570 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:21.892797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.892775 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:45:21.899432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.899407 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5d92fa-5497549574-m64fx"] Oct 30 19:45:21.939372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.939337 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\") pod \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " Oct 30 19:45:21.939884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.939837 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca" pod "f1c7380e-7d52-4b6a-b196-278c1f53bf15" (UID: "f1c7380e-7d52-4b6a-b196-278c1f53bf15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.940250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.940224 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\") pod \"729ea970-3ebb-41fa-8c70-26ed3270af88\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " Oct 30 19:45:21.940324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.940277 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\" (UniqueName: \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\") pod \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\" (UID: \"f1c7380e-7d52-4b6a-b196-278c1f53bf15\") " Oct 30 19:45:21.940409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.940394 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\" (UniqueName: \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\") pod \"729ea970-3ebb-41fa-8c70-26ed3270af88\" (UID: \"729ea970-3ebb-41fa-8c70-26ed3270af88\") " Oct 30 19:45:21.940499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.940481 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\") pod \"a330ce4c-4834-4b6e-94af-b2370c7273b3\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " Oct 30 19:45:21.940719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.940703 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\" (UniqueName: \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\") pod \"a330ce4c-4834-4b6e-94af-b2370c7273b3\" (UID: \"a330ce4c-4834-4b6e-94af-b2370c7273b3\") " Oct 30 19:45:21.941372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.941346 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c7380e-7d52-4b6a-b196-278c1f53bf15-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.941627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.941602 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca" pod "729ea970-3ebb-41fa-8c70-26ed3270af88" (UID: "729ea970-3ebb-41fa-8c70-26ed3270af88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.941846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.941818 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca" pod "a330ce4c-4834-4b6e-94af-b2370c7273b3" (UID: "a330ce4c-4834-4b6e-94af-b2370c7273b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.945288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.945256 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr" pod "729ea970-3ebb-41fa-8c70-26ed3270af88" (UID: "729ea970-3ebb-41fa-8c70-26ed3270af88"). InnerVolumeSpecName "kube-api-access-b4slr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.946302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.946272 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t" pod "f1c7380e-7d52-4b6a-b196-278c1f53bf15" (UID: "f1c7380e-7d52-4b6a-b196-278c1f53bf15"). InnerVolumeSpecName "kube-api-access-zxb9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.946376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:21.946294 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp" pod "a330ce4c-4834-4b6e-94af-b2370c7273b3" (UID: "a330ce4c-4834-4b6e-94af-b2370c7273b3"). InnerVolumeSpecName "kube-api-access-drpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:22.043490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.043406 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/729ea970-3ebb-41fa-8c70-26ed3270af88-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.043490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.043445 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxb9t\" (UniqueName: \"kubernetes.io/projected/f1c7380e-7d52-4b6a-b196-278c1f53bf15-kube-api-access-zxb9t\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.043490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.043462 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4slr\" (UniqueName: \"kubernetes.io/projected/729ea970-3ebb-41fa-8c70-26ed3270af88-kube-api-access-b4slr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.043490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.043477 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a330ce4c-4834-4b6e-94af-b2370c7273b3-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.043490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.043490 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drpmp\" (UniqueName: \"kubernetes.io/projected/a330ce4c-4834-4b6e-94af-b2370c7273b3-kube-api-access-drpmp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.054363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.054330 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:45:22.088859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.088826 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:45:22.244591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.244550 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.410463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.409362 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a8917f-6024-4417-a379-02623868b1eb" path="/var/lib/kubelet/pods/16a8917f-6024-4417-a379-02623868b1eb/volumes" Oct 30 19:45:22.410463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.410138 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46245223-da25-4bae-89bb-fa85f68514a1" path="/var/lib/kubelet/pods/46245223-da25-4bae-89bb-fa85f68514a1/volumes" Oct 30 19:45:22.411795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.411136 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" path="/var/lib/kubelet/pods/48e02b96-75a1-48fc-a40a-e705ef4c8ddc/volumes" Oct 30 19:45:22.414567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.412414 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" path="/var/lib/kubelet/pods/66bc570f-670a-4b7c-b71b-ab3a7914a823/volumes" Oct 30 19:45:22.414567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.413189 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" path="/var/lib/kubelet/pods/826b31fe-33ad-4c84-8750-465f77bb2c25/volumes" Oct 30 19:45:22.414567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.413872 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" path="/var/lib/kubelet/pods/8fa19767-d08d-49d4-a2bb-d4d1302879e9/volumes" Oct 30 19:45:22.417102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.417062 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" path="/var/lib/kubelet/pods/aa571faa-7710-4202-9bb0-2eaf0b1312e9/volumes" Oct 30 19:45:22.417793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.417770 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" path="/var/lib/kubelet/pods/b08de27d-a43f-4763-9c03-391b03f1a848/volumes" Oct 30 19:45:22.420749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.420567 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.578863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.578834 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" Oct 30 19:45:22.578863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.578853 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz" event={"ID":"729ea970-3ebb-41fa-8c70-26ed3270af88","Type":"ContainerDied","Data":"70a37d9d0d7bd333dce33b8cf5b2697198a5fae04929f63e90b1be723b77758f"} Oct 30 19:45:22.579366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.578924 4990 scope.go:122] "RemoveContainer" containerID="9453327b650ed9e214fe70d363cc041122924fa1a4b0507240aff515378793a3" Oct 30 19:45:22.582029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.582006 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" Oct 30 19:45:22.582217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.582026 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp" event={"ID":"a330ce4c-4834-4b6e-94af-b2370c7273b3","Type":"ContainerDied","Data":"78807b92ed61de5062edbfb65a811da93b4addee7a1a861e609f27e568c29889"} Oct 30 19:45:22.584857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.584302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" event={"ID":"f1c7380e-7d52-4b6a-b196-278c1f53bf15","Type":"ContainerDied","Data":"bf614b3bc4e39bcb81b924c97c2564011a3f4695d9b872d8dfba1f16aca88e96"} Oct 30 19:45:22.584857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.584342 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5" Oct 30 19:45:22.594678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.594662 4990 scope.go:122] "RemoveContainer" containerID="0ea09696fb3423b74626fed68b1086b4623d8319946d7e4264225d1edda8f2ea" Oct 30 19:45:22.604458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.604439 4990 scope.go:122] "RemoveContainer" containerID="02228bd0e506c510d01d55ec170e49c24d99b154796e22caaf12fee414261f89" Oct 30 19:45:22.628427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.628402 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:22.634132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.634109 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-5n8gz"] Oct 30 19:45:22.666162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.666068 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:22.669745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.669719 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-576a62-29889825-xz6l5"] Oct 30 19:45:22.718114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.714654 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:22.718114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.717500 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889825-jw4lp"] Oct 30 19:45:22.996982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:22.996898 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.379808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:24.379771 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:24.402946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:24.402921 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" path="/var/lib/kubelet/pods/729ea970-3ebb-41fa-8c70-26ed3270af88/volumes" Oct 30 19:45:24.403507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:24.403491 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" path="/var/lib/kubelet/pods/a330ce4c-4834-4b6e-94af-b2370c7273b3/volumes" Oct 30 19:45:24.403942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:24.403928 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" path="/var/lib/kubelet/pods/f1c7380e-7d52-4b6a-b196-278c1f53bf15/volumes" Oct 30 19:45:24.958725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:24.958660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.200396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:25.200323 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.387118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:25.383988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.587810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:25.587775 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.791674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:25.791578 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.054620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:27.054562 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" containerID="cri-o://77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf" gracePeriod=25 Oct 30 19:45:27.089684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:27.089648 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" containerID="cri-o://29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c" gracePeriod=25 Oct 30 19:45:27.185488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:27.185453 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.494236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:27.494151 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:45:27.494410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:27.494366 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" containerID="cri-o://ae5bb4a7bcc637c0bdf5086128fe109a3fa9bb6e9e5bb5be1b5b0b5642ab69d0" gracePeriod=30 Oct 30 19:45:28.910802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:28.910766 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.492081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:30.492043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:31.252806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:31.252766 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:31.253003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:31.252766 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:31.488540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:31.488499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.211697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.211643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.448540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.448503 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.608788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.608753 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.718173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.718143 4990 generic.go:356] "Generic (PLEG): container finished" podID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerID="ae5bb4a7bcc637c0bdf5086128fe109a3fa9bb6e9e5bb5be1b5b0b5642ab69d0" exitCode=0 Oct 30 19:45:32.718325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.718211 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" event={"ID":"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d","Type":"ContainerDied","Data":"ae5bb4a7bcc637c0bdf5086128fe109a3fa9bb6e9e5bb5be1b5b0b5642ab69d0"} Oct 30 19:45:32.803658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.803630 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:45:32.861012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.860977 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.943969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.943938 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.943983 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944012 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944046 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944140 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944203 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944285 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.944438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944294 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944377 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944522 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.944677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944538 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\") pod \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\" (UID: \"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d\") " Oct 30 19:45:32.944677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.944867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944844 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:32.944867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.944850 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.945415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.945389 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:32.945593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.945420 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:32.945593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.945436 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:32.945593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.945449 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:32.945593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.945461 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:32.946417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.946395 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.946584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.946555 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:32.946716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.946657 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "kube-api-access-gnhw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:32.946716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:32.946695 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token" pod "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" (UID: "4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:33.058142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.056359 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:33.058142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.056400 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhw8\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-kube-api-access-gnhw8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:33.058142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.056416 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:33.058142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.056430 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:33.727719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.727688 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" Oct 30 19:45:33.728153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.727722 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv" event={"ID":"4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d","Type":"ContainerDied","Data":"b93575377fbbbcc7a01cefd7a717600bc995e71d8610c9b326af91609167a7de"} Oct 30 19:45:33.728153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.727791 4990 scope.go:122] "RemoveContainer" containerID="ae5bb4a7bcc637c0bdf5086128fe109a3fa9bb6e9e5bb5be1b5b0b5642ab69d0" Oct 30 19:45:33.787232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.787201 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:45:33.794738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:33.794707 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7hbfv"] Oct 30 19:45:34.174002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:34.173955 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:34.361903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:34.361818 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:34.407136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:34.407084 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" path="/var/lib/kubelet/pods/4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d/volumes" Oct 30 19:45:34.578820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:34.578784 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:35.097579 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:35.097535 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:45:36.013113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:36.011191 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:37.702017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:37.699286 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.399744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:39.399703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:41.017993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:41.017931 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:41.252562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:41.252528 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:41.252708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:41.252530 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:45:42.696143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:42.694888 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.591273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:44.590774 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:46.225600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:46.225530 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.398154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.398125 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:45:47.432607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.432573 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:45:47.501938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.501899 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\") pod \"e664fa9a-5236-445a-9d95-5fa23503dbb6\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " Oct 30 19:45:47.502161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.502056 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\" (UniqueName: \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\") pod \"e664fa9a-5236-445a-9d95-5fa23503dbb6\" (UID: \"e664fa9a-5236-445a-9d95-5fa23503dbb6\") " Oct 30 19:45:47.502161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.502122 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\") pod \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " Oct 30 19:45:47.502390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.502367 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\" (UniqueName: \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\") pod \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\" (UID: \"dcb083da-c973-4f04-baff-a9e4eb0a6c1c\") " Oct 30 19:45:47.502765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.502720 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume" pod "dcb083da-c973-4f04-baff-a9e4eb0a6c1c" (UID: "dcb083da-c973-4f04-baff-a9e4eb0a6c1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:47.502765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.502750 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume" pod "e664fa9a-5236-445a-9d95-5fa23503dbb6" (UID: "e664fa9a-5236-445a-9d95-5fa23503dbb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:47.503046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.503021 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e664fa9a-5236-445a-9d95-5fa23503dbb6-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.503216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.503050 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.505338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.505303 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch" pod "e664fa9a-5236-445a-9d95-5fa23503dbb6" (UID: "e664fa9a-5236-445a-9d95-5fa23503dbb6"). InnerVolumeSpecName "kube-api-access-4ncch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:47.505592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.505568 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b" pod "dcb083da-c973-4f04-baff-a9e4eb0a6c1c" (UID: "dcb083da-c973-4f04-baff-a9e4eb0a6c1c"). InnerVolumeSpecName "kube-api-access-2vm2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:47.604990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.604902 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vm2b\" (UniqueName: \"kubernetes.io/projected/dcb083da-c973-4f04-baff-a9e4eb0a6c1c-kube-api-access-2vm2b\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.604990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.604939 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ncch\" (UniqueName: \"kubernetes.io/projected/e664fa9a-5236-445a-9d95-5fa23503dbb6-kube-api-access-4ncch\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:45:47.925635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.923024 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.950958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.949328 4990 generic.go:356] "Generic (PLEG): container finished" podID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerID="29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c" exitCode=0 Oct 30 19:45:47.950958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.949655 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" event={"ID":"dcb083da-c973-4f04-baff-a9e4eb0a6c1c","Type":"ContainerDied","Data":"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c"} Oct 30 19:45:47.950958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.949712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" event={"ID":"dcb083da-c973-4f04-baff-a9e4eb0a6c1c","Type":"ContainerDied","Data":"359449f7db8fb1409cf9126231661254198ad0db1795d25ebd6f8318daf257b6"} Oct 30 19:45:47.950958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.949745 4990 scope.go:122] "RemoveContainer" containerID="29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c" Oct 30 19:45:47.950958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.949913 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842" Oct 30 19:45:47.967387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.966749 4990 generic.go:356] "Generic (PLEG): container finished" podID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerID="77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf" exitCode=0 Oct 30 19:45:47.967387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.966797 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" Oct 30 19:45:47.967387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.966921 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" event={"ID":"e664fa9a-5236-445a-9d95-5fa23503dbb6","Type":"ContainerDied","Data":"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf"} Oct 30 19:45:47.967387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.966962 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9" event={"ID":"e664fa9a-5236-445a-9d95-5fa23503dbb6","Type":"ContainerDied","Data":"fd073a287c688ba0fbe41c0010fc3b7bb6b50096c4ed5e19f4bb24ccb0fa331c"} Oct 30 19:45:47.984522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.984497 4990 scope.go:122] "RemoveContainer" containerID="29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c" Oct 30 19:45:47.984893 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:47.984857 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c\": container with ID starting with 29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c not found: ID does not exist" containerID="29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c" Oct 30 19:45:47.985003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.984905 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c"} err="failed to get container status \"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c\": rpc error: code = NotFound desc = could not find container \"29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c\": container with ID starting with 29467d34c647df520a338e29627f859200d4a972f8c71481d7fead97b9d1623c not found: ID does not exist" Oct 30 19:45:47.985003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.984934 4990 scope.go:122] "RemoveContainer" containerID="77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf" Oct 30 19:45:47.999193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.999171 4990 scope.go:122] "RemoveContainer" containerID="77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf" Oct 30 19:45:47.999466 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:45:47.999445 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf\": container with ID starting with 77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf not found: ID does not exist" containerID="77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf" Oct 30 19:45:47.999547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:47.999478 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf"} err="failed to get container status \"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf\": rpc error: code = NotFound desc = could not find container \"77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf\": container with ID starting with 77ec898935dd76b674e4637516c47d4ac62f1d72eabddd8f877b0e14b9e9d8bf not found: ID does not exist" Oct 30 19:45:48.019698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.019669 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:45:48.030686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.030664 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-469724-5b9fbc46c8-7txc9"] Oct 30 19:45:48.055143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.055121 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:45:48.058880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.058862 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-469724-9fd448b79-hh842"] Oct 30 19:45:48.402916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.402885 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" path="/var/lib/kubelet/pods/dcb083da-c973-4f04-baff-a9e4eb0a6c1c/volumes" Oct 30 19:45:48.403380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:48.403363 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" path="/var/lib/kubelet/pods/e664fa9a-5236-445a-9d95-5fa23503dbb6/volumes" Oct 30 19:45:49.790593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:49.790540 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.523755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:51.523720 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:53.141422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:53.141385 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:54.742881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:54.742800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.019055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:56.019016 4990 scope.go:122] "RemoveContainer" containerID="9b520886d9a403b733e3eb80f6d44b6bccccc7e2dd918990f59ea27af63ad0d9" Oct 30 19:45:56.522376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:56.522342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.904958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:56.904920 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 19:45:58.247641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:58.247604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:59.917337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:45:59.917287 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.020164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:01.020120 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.414147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:01.414081 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.730188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:01.730070 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:03.335310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:03.335271 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:04.938625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:04.938588 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:05.099301 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:46:05.099261 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:46:06.652796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:06.652759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:07.115555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:07.115517 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:08.366139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:08.366064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:08.507722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:08.507684 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:09.998656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:09.998617 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:11.857835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:11.857802 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:13.427453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:13.427420 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:15.321049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:15.315891 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:17.027006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:17.026965 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:17.502578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:17.502529 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:18.866078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:18.866035 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:20.641828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:20.641778 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.031274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:21.031189 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:22.482978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:22.482946 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:24.111722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:24.111637 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:25.822251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:25.822212 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.019989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:26.019955 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:46:26.205992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:26.205903 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.503534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:27.503492 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:29.634873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:29.634829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.112410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:30.112370 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.850116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:31.847083 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.496642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:33.496588 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:35.098348 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:46:35.098309 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:46:35.139569 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:35.139504 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:36.095874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:36.095840 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:36.989399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:36.989360 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.699332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:38.699292 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:40.340904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:40.340858 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:41.046104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.046056 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" podUID="21221a17-6537-473a-86b5-28f1786f7271" containerName="main" containerID="cri-o://ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025" gracePeriod=45 Oct 30 19:46:41.329370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.329344 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:46:41.389965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.389936 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\" (UniqueName: \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.390677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.390654 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.390694 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.390814 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\" (UniqueName: \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.390990 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391043 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\") pod \"21221a17-6537-473a-86b5-28f1786f7271\" (UID: \"21221a17-6537-473a-86b5-28f1786f7271\") " Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391603 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391667 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391763 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391782 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.391846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.391795 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:41.395057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.394482 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:41.396273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.396250 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "kube-api-access-85p72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:41.396375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.396324 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm" pod "21221a17-6537-473a-86b5-28f1786f7271" (UID: "21221a17-6537-473a-86b5-28f1786f7271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:41.484916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.484630 4990 generic.go:356] "Generic (PLEG): container finished" podID="21221a17-6537-473a-86b5-28f1786f7271" containerID="ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025" exitCode=0 Oct 30 19:46:41.484916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.484709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" event={"ID":"21221a17-6537-473a-86b5-28f1786f7271","Type":"ContainerDied","Data":"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025"} Oct 30 19:46:41.484916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.484762 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" event={"ID":"21221a17-6537-473a-86b5-28f1786f7271","Type":"ContainerDied","Data":"a5394deb5a64e63d83d460b9306efabbf6665b914bc22460cfe779d4b01b2382"} Oct 30 19:46:41.484916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.484785 4990 scope.go:122] "RemoveContainer" containerID="ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025" Oct 30 19:46:41.485905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.485888 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv" Oct 30 19:46:41.496053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.494472 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.496053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.494780 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85p72\" (UniqueName: \"kubernetes.io/projected/21221a17-6537-473a-86b5-28f1786f7271-kube-api-access-85p72\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.496053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.494798 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21221a17-6537-473a-86b5-28f1786f7271-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.496053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.494812 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21221a17-6537-473a-86b5-28f1786f7271-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:41.515257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.515234 4990 scope.go:122] "RemoveContainer" containerID="ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025" Oct 30 19:46:41.515606 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:46:41.515584 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025\": container with ID starting with ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025 not found: ID does not exist" containerID="ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025" Oct 30 19:46:41.515703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.515611 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025"} err="failed to get container status \"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025\": rpc error: code = NotFound desc = could not find container \"ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025\": container with ID starting with ea32ed9a556f79cc5926f784f72bf25a8597640e5b605a843e4be038c8f74025 not found: ID does not exist" Oct 30 19:46:41.544162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.544130 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:46:41.547974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.547956 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/auth-test-model-6b7ccc-kserve-bdf44f8df-qdmcv"] Oct 30 19:46:41.938933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:41.938874 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:42.413469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:42.413430 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21221a17-6537-473a-86b5-28f1786f7271" path="/var/lib/kubelet/pods/21221a17-6537-473a-86b5-28f1786f7271/volumes" Oct 30 19:46:43.429398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:43.429356 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:46:43.519653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:43.519618 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:43.661962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:43.661923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.240428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:45.240396 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:47.263980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:47.263941 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:49.063703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:49.063642 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.392713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:50.392680 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:51.093903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:51.093853 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.833782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:52.833742 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:54.445305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:54.443121 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:56.235710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:56.235670 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:56.534472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:56.534380 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:56.689306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:56.689264 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.024340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:58.024105 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.469025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:58.468980 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" podUID="9c74e543-3323-481f-b645-78bcb707f3e6" containerName="main" containerID="cri-o://8fccc4b433775bc46db2b7382f054941b92123f8b1e0c216dee526f2a12656d4" gracePeriod=45 Oct 30 19:46:58.642762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:58.642728 4990 generic.go:356] "Generic (PLEG): container finished" podID="9c74e543-3323-481f-b645-78bcb707f3e6" containerID="8fccc4b433775bc46db2b7382f054941b92123f8b1e0c216dee526f2a12656d4" exitCode=0 Oct 30 19:46:58.642999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:58.642917 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" event={"ID":"9c74e543-3323-481f-b645-78bcb707f3e6","Type":"ContainerDied","Data":"8fccc4b433775bc46db2b7382f054941b92123f8b1e0c216dee526f2a12656d4"} Oct 30 19:46:59.030163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.029522 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:46:59.111119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.110843 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.111119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.110909 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\" (UniqueName: \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.111119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.110937 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.111119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.111072 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.111559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.111234 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.111559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.111387 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\") pod \"9c74e543-3323-481f-b645-78bcb707f3e6\" (UID: \"9c74e543-3323-481f-b645-78bcb707f3e6\") " Oct 30 19:46:59.115113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.112431 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:59.115113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.112649 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:59.115113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.113609 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:59.146749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.144821 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:59.160903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.160798 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:59.161111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.160949 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82" pod "9c74e543-3323-481f-b645-78bcb707f3e6" (UID: "9c74e543-3323-481f-b645-78bcb707f3e6"). InnerVolumeSpecName "kube-api-access-h5g82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:59.214388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.214345 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.214656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.214637 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.214788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.214772 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5g82\" (UniqueName: \"kubernetes.io/projected/9c74e543-3323-481f-b645-78bcb707f3e6-kube-api-access-h5g82\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.214913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.214897 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.215022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.215008 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c74e543-3323-481f-b645-78bcb707f3e6-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.215165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.215151 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c74e543-3323-481f-b645-78bcb707f3e6-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:46:59.654660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.654623 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" event={"ID":"9c74e543-3323-481f-b645-78bcb707f3e6","Type":"ContainerDied","Data":"5b16903527337d1f88f8e191b11bc9d2c91717fdabdec7319db1e855e4bb7f33"} Oct 30 19:46:59.654660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.654646 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr" Oct 30 19:46:59.654918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.654677 4990 scope.go:122] "RemoveContainer" containerID="8fccc4b433775bc46db2b7382f054941b92123f8b1e0c216dee526f2a12656d4" Oct 30 19:46:59.709153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.709010 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:46:59.723990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:46:59.723951 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-614c02/test-model-614c02-kserve-8c5d7c9b7-tbzpr"] Oct 30 19:47:00.334999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:00.334954 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.407886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:00.407848 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c74e543-3323-481f-b645-78bcb707f3e6" path="/var/lib/kubelet/pods/9c74e543-3323-481f-b645-78bcb707f3e6/volumes" Oct 30 19:47:01.151617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:01.151570 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.310743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:01.310702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.558182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:01.558148 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.175999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:02.175944 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.764621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:03.764581 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:04.665822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:04.665702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:05.105748 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:47:05.105697 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:47:05.520401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:05.520306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:06.422268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:06.422226 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.193123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:07.192980 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.731385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:07.731328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:08.852048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:08.852008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:10.544606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:10.544559 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.187694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:12.187654 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:13.852997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:13.852956 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:15.551177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:15.551142 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.095869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:17.095828 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.706122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:18.705389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:20.558648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:20.558568 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.011933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:22.011896 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.709164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:22.709128 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.887758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:22.887715 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.041804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:23.041716 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.388274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:23.388197 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.881400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:23.881361 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.916173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:24.910875 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.285530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:26.285489 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.935310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:26.935270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.465818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:28.465773 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.645118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:28.644476 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:29.472340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:29.472306 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:47:29.627119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:29.626788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.122686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.122653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.272692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.272640 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.509445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.509337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.526204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.521393 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:47:30.526204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.521594 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" containerID="cri-o://62ccdc5abb2d3740ccb1a98384f8ae50bc249c6cfd00862c19086d11599e9135" gracePeriod=30 Oct 30 19:47:30.735245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.735209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.892116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:30.891223 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.960297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:31.960258 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:32.105852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:32.105813 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:32.244614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:32.244505 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.736500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:33.735335 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.483937 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:34.483900 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:47:34.513769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:34.513738 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:47:34.670613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:34.670580 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.823442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:34.823408 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:35.125169 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:47:35.118349 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:47:35.621721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:35.621685 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:36.107858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.107821 4990 generic.go:356] "Generic (PLEG): container finished" podID="63af8504-30b6-435e-a04c-5e14cac99c80" containerID="62ccdc5abb2d3740ccb1a98384f8ae50bc249c6cfd00862c19086d11599e9135" exitCode=0 Oct 30 19:47:36.108347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.107983 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" event={"ID":"63af8504-30b6-435e-a04c-5e14cac99c80","Type":"ContainerDied","Data":"62ccdc5abb2d3740ccb1a98384f8ae50bc249c6cfd00862c19086d11599e9135"} Oct 30 19:47:36.232167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.232137 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:47:36.250239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250205 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250375 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250421 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250438 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250507 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250593 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250653 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250738 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250769 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\") pod \"63af8504-30b6-435e-a04c-5e14cac99c80\" (UID: \"63af8504-30b6-435e-a04c-5e14cac99c80\") " Oct 30 19:47:36.250890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250836 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:36.250945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250911 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:36.251009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.250947 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:36.251177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251150 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:36.251281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251209 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:36.251495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251479 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63af8504-30b6-435e-a04c-5e14cac99c80-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.251555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251497 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.251555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251506 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.251555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251514 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.251555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.251528 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.253131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.253104 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:36.253227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.253163 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "kube-api-access-xsk5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:36.253464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.253436 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:36.253464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.253443 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token" pod "63af8504-30b6-435e-a04c-5e14cac99c80" (UID: "63af8504-30b6-435e-a04c-5e14cac99c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:36.353270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.353233 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63af8504-30b6-435e-a04c-5e14cac99c80-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.353270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.353266 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.353270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.353278 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63af8504-30b6-435e-a04c-5e14cac99c80-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.353533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:36.353290 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsk5h\" (UniqueName: \"kubernetes.io/projected/63af8504-30b6-435e-a04c-5e14cac99c80-kube-api-access-xsk5h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:37.120072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.120043 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" Oct 30 19:47:37.120567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.120075 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4" event={"ID":"63af8504-30b6-435e-a04c-5e14cac99c80","Type":"ContainerDied","Data":"b151df0fef493d569ed49a3ad6d4e52e74121bc1f238d78709d6c3ca0ff08f14"} Oct 30 19:47:37.120567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.120151 4990 scope.go:122] "RemoveContainer" containerID="62ccdc5abb2d3740ccb1a98384f8ae50bc249c6cfd00862c19086d11599e9135" Oct 30 19:47:37.178359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.178329 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:47:37.189144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.189111 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-gbft4"] Oct 30 19:47:37.292329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:37.292294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:38.404475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:38.404433 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" path="/var/lib/kubelet/pods/63af8504-30b6-435e-a04c-5e14cac99c80/volumes" Oct 30 19:47:39.025477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:39.025436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.485242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:39.485197 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" containerID="cri-o://2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc" gracePeriod=25 Oct 30 19:47:39.514040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:39.513965 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" containerID="cri-o://c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce" gracePeriod=25 Oct 30 19:47:40.735759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:40.732395 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:42.544848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:42.544812 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.787873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:43.787834 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:43.788336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:43.787958 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:44.312273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.312216 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.494152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.494072 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" podUID="417cd409-7aa8-4d53-a675-adb6a271e717" containerName="main" containerID="cri-o://c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae" gracePeriod=45 Oct 30 19:47:44.501461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.501426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.671311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.671270 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.820562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.820533 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:47:44.879426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879315 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\" (UniqueName: \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.879607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879488 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.879607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879553 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\" (UniqueName: \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.879738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879633 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.879738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879690 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.879850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.879798 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\") pod \"417cd409-7aa8-4d53-a675-adb6a271e717\" (UID: \"417cd409-7aa8-4d53-a675-adb6a271e717\") " Oct 30 19:47:44.881216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.881164 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:44.881402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.881338 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:44.881532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.881508 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:44.884535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.884489 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "kube-api-access-hwr99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:44.885142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.885083 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:44.905383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.904781 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm" pod "417cd409-7aa8-4d53-a675-adb6a271e717" (UID: "417cd409-7aa8-4d53-a675-adb6a271e717"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:44.982146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982041 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwr99\" (UniqueName: \"kubernetes.io/projected/417cd409-7aa8-4d53-a675-adb6a271e717-kube-api-access-hwr99\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:44.982146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982107 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:44.982146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982125 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/417cd409-7aa8-4d53-a675-adb6a271e717-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:44.982146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982142 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:44.982146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982154 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:44.982595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:44.982166 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417cd409-7aa8-4d53-a675-adb6a271e717-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:45.191176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.191045 4990 generic.go:356] "Generic (PLEG): container finished" podID="417cd409-7aa8-4d53-a675-adb6a271e717" containerID="c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae" exitCode=0 Oct 30 19:47:45.191176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.191141 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" Oct 30 19:47:45.191176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.191148 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" event={"ID":"417cd409-7aa8-4d53-a675-adb6a271e717","Type":"ContainerDied","Data":"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae"} Oct 30 19:47:45.191472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.191200 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482" event={"ID":"417cd409-7aa8-4d53-a675-adb6a271e717","Type":"ContainerDied","Data":"d19005fa1578ec45cc274f6f8df73efcf8f3941b24426ae452b0c096135b3c91"} Oct 30 19:47:45.191472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.191223 4990 scope.go:122] "RemoveContainer" containerID="c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae" Oct 30 19:47:45.204429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.204403 4990 scope.go:122] "RemoveContainer" containerID="c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae" Oct 30 19:47:45.204840 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:47:45.204809 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae\": container with ID starting with c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae not found: ID does not exist" containerID="c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae" Oct 30 19:47:45.204957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.204850 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae"} err="failed to get container status \"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae\": rpc error: code = NotFound desc = could not find container \"c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae\": container with ID starting with c6fa95c5f68aecc737474e5d96aa3b2637c416c166d297efbb9c6bea9be554ae not found: ID does not exist" Oct 30 19:47:45.237333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.237288 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:47:45.242259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:45.242223 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f44f74/auth-test-model-f44f74-kserve-d9dd74965-q7482"] Oct 30 19:47:46.222121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:46.219583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:46.406383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:46.406344 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417cd409-7aa8-4d53-a675-adb6a271e717" path="/var/lib/kubelet/pods/417cd409-7aa8-4d53-a675-adb6a271e717/volumes" Oct 30 19:47:46.986939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:46.986903 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:47.032921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.032870 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:47:47.034077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034049 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.034163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034082 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.034163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034118 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.034163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034126 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.034163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034145 4990 cpu_manager.go:414] "containerMap: removing container" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034153 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034166 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034174 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034193 4990 cpu_manager.go:414] "containerMap: removing container" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034200 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034216 4990 cpu_manager.go:414] "containerMap: removing container" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034223 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034233 4990 cpu_manager.go:414] "containerMap: removing container" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034240 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034249 4990 cpu_manager.go:414] "containerMap: removing container" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034256 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034274 4990 cpu_manager.go:414] "containerMap: removing container" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034281 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034296 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9c74e543-3323-481f-b645-78bcb707f3e6" containerName="main" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034303 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c74e543-3323-481f-b645-78bcb707f3e6" containerName="main" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034313 4990 cpu_manager.go:414] "containerMap: removing container" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034321 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034335 4990 cpu_manager.go:414] "containerMap: removing container" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034344 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034355 4990 cpu_manager.go:414] "containerMap: removing container" podUID="417cd409-7aa8-4d53-a675-adb6a271e717" containerName="main" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034362 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417cd409-7aa8-4d53-a675-adb6a271e717" containerName="main" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034374 4990 cpu_manager.go:414] "containerMap: removing container" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034382 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034392 4990 cpu_manager.go:414] "containerMap: removing container" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034398 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034412 4990 cpu_manager.go:414] "containerMap: removing container" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034418 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034429 4990 cpu_manager.go:414] "containerMap: removing container" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034436 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034452 4990 cpu_manager.go:414] "containerMap: removing container" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034461 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034472 4990 cpu_manager.go:414] "containerMap: removing container" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034479 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034489 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034496 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034505 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034512 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034523 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034530 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034542 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034549 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034566 4990 cpu_manager.go:414] "containerMap: removing container" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034572 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034587 4990 cpu_manager.go:414] "containerMap: removing container" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034594 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034616 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034624 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034641 4990 cpu_manager.go:414] "containerMap: removing container" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034647 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034668 4990 cpu_manager.go:414] "containerMap: removing container" podUID="21221a17-6537-473a-86b5-28f1786f7271" containerName="main" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034678 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21221a17-6537-473a-86b5-28f1786f7271" containerName="main" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034688 4990 cpu_manager.go:414] "containerMap: removing container" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034695 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034710 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034717 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034728 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034734 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034750 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034758 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034770 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f0029510-1083-4fec-935d-03d20bcdda8b" containerName="maas-api" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034777 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0029510-1083-4fec-935d-03d20bcdda8b" containerName="maas-api" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034787 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034794 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034805 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034813 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034830 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034837 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034854 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034860 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034869 4990 cpu_manager.go:414] "containerMap: removing container" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034876 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034894 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.034901 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035081 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c74e543-3323-481f-b645-78bcb707f3e6" containerName="main" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035120 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035132 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="158ecac9-d0f1-4aa7-b714-04b49ea96832" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035141 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="417cd409-7aa8-4d53-a675-adb6a271e717" containerName="main" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035149 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035157 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035165 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035174 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035185 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035193 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035200 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035209 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035217 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035225 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035234 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="63af8504-30b6-435e-a04c-5e14cac99c80" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035246 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9e8c5d-ad6d-4f04-88d8-fc116dd4711d" containerName="istio-proxy" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035256 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035264 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035273 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035282 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="21221a17-6537-473a-86b5-28f1786f7271" containerName="main" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035290 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035300 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e664fa9a-5236-445a-9d95-5fa23503dbb6" containerName="payload-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035313 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035322 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0029510-1083-4fec-935d-03d20bcdda8b" containerName="maas-api" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035333 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035342 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa19767-d08d-49d4-a2bb-d4d1302879e9" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035352 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035363 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb083da-c973-4f04-baff-a9e4eb0a6c1c" containerName="payload-pre-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035372 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035381 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035389 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035399 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035409 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08de27d-a43f-4763-9c03-391b03f1a848" containerName="payload-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035417 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a8917f-6024-4417-a379-02623868b1eb" containerName="payload-pre-processing" Oct 30 19:47:47.035390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035427 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.037916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035437 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.037916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.035448 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="46245223-da25-4bae-89bb-fa85f68514a1" containerName="cleanup" Oct 30 19:47:47.041563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.041534 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.064317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.064275 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:47:47.068263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.068224 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54086b\"" Oct 30 19:47:47.109041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.108988 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:47:47.110247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110188 4990 cpu_manager.go:414] "containerMap: removing container" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.110247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110218 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa571faa-7710-4202-9bb0-2eaf0b1312e9" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110297 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110308 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110333 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110341 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330ce4c-4834-4b6e-94af-b2370c7273b3" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110371 4990 cpu_manager.go:414] "containerMap: removing container" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110379 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110393 4990 cpu_manager.go:414] "containerMap: removing container" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.110503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110401 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110578 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c7380e-7d52-4b6a-b196-278c1f53bf15" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110597 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e02b96-75a1-48fc-a40a-e705ef4c8ddc" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110608 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="729ea970-3ebb-41fa-8c70-26ed3270af88" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110620 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6f7330-efcc-4af9-94f6-c46fdce04047" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110630 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9cf908-cf47-4206-aa75-fcf4ed2aa937" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110646 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="826b31fe-33ad-4c84-8750-465f77bb2c25" containerName="cleanup" Oct 30 19:47:47.110923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.110656 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bc570f-670a-4b7c-b71b-ab3a7914a823" containerName="cleanup" Oct 30 19:47:47.115221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.114832 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.115221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.115014 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxsj\" (UniqueName: \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.120449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.120410 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.144306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.142716 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:47:47.144306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.144282 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54086b-dockercfg-9mn8n\"" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.241442 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkhh\" (UniqueName: \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.241560 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.241615 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.241818 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxsj\" (UniqueName: \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.243518 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.269723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.244147 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:47.347955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.346004 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkhh\" (UniqueName: \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.347955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.346114 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.347955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.347134 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.349574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.349511 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxsj\" (UniqueName: \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\") pod \"payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.362664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.358283 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:47.406319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.387535 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkhh\" (UniqueName: \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\") pod \"payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.447602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.447567 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:47.725604 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:47:47.725567 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237f02be_7d7d_4b1c_bc7d_e6f6f77f0c4c.slice/crio-447406b9e3ff9a02b5561d6433e29b986f738a0c6fe8b3ecaef7239c4e81b082 WatchSource:0}: Error finding container 447406b9e3ff9a02b5561d6433e29b986f738a0c6fe8b3ecaef7239c4e81b082: Status 404 returned error can't find the container with id 447406b9e3ff9a02b5561d6433e29b986f738a0c6fe8b3ecaef7239c4e81b082 Oct 30 19:47:47.726245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.726217 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:47:47.804059 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:47:47.804031 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedadb80_d646_4b09_abd3_23cdafab6bf0.slice/crio-a493527b9a08d9a24f1e0e5d03ec545df81c6aed2e2fdeed635d7ab30359dd9f WatchSource:0}: Error finding container a493527b9a08d9a24f1e0e5d03ec545df81c6aed2e2fdeed635d7ab30359dd9f: Status 404 returned error can't find the container with id a493527b9a08d9a24f1e0e5d03ec545df81c6aed2e2fdeed635d7ab30359dd9f Oct 30 19:47:47.804431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:47.804364 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:47:48.229895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.229786 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:48.320964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.320910 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" event={"ID":"dedadb80-d646-4b09-abd3-23cdafab6bf0","Type":"ContainerStarted","Data":"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390"} Oct 30 19:47:48.321518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.320987 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" event={"ID":"dedadb80-d646-4b09-abd3-23cdafab6bf0","Type":"ContainerStarted","Data":"a493527b9a08d9a24f1e0e5d03ec545df81c6aed2e2fdeed635d7ab30359dd9f"} Oct 30 19:47:48.321518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.321011 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:48.323331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.323299 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" event={"ID":"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c","Type":"ContainerStarted","Data":"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7"} Oct 30 19:47:48.323460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.323354 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" event={"ID":"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c","Type":"ContainerStarted","Data":"447406b9e3ff9a02b5561d6433e29b986f738a0c6fe8b3ecaef7239c4e81b082"} Oct 30 19:47:48.323460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.323422 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:48.361884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.361823 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" podStartSLOduration=1.361803278 podStartE2EDuration="1.361803278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:48.354917237 +0000 UTC m=+2576.066442604" watchObservedRunningTime="2026-10-30 19:47:48.361803278 +0000 UTC m=+2576.073328653" Oct 30 19:47:48.426183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:48.425203 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" podStartSLOduration=2.425177453 podStartE2EDuration="2.425177453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:48.406359949 +0000 UTC m=+2576.117885328" watchObservedRunningTime="2026-10-30 19:47:48.425177453 +0000 UTC m=+2576.136702820" Oct 30 19:47:50.277177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:50.277009 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.548157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:51.548115 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:47:51.695944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:51.695900 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.836828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:51.836786 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.039501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:52.039447 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:53.437615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:53.437577 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:53.794837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:53.794739 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:53.799734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:53.799686 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:54.124508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:54.124469 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.279170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:54.278990 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.743762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:54.743723 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:56.150577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:56.150529 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:56.548565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:56.548465 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" containerID="cri-o://3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912" gracePeriod=25 Oct 30 19:47:56.905578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:56.905537 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:57.792398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:57.782178 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:59.358082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.358054 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:47:59.358589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.358156 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:47:59.431422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.431386 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:59.828915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.828725 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:47:59.885461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.885354 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:47:59.943772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.943734 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\" (UniqueName: \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\") pod \"19088b0a-79fa-4237-af46-4ed8847e00df\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " Oct 30 19:47:59.944158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.944133 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\") pod \"19088b0a-79fa-4237-af46-4ed8847e00df\" (UID: \"19088b0a-79fa-4237-af46-4ed8847e00df\") " Oct 30 19:47:59.944393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.944362 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\" (UniqueName: \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\") pod \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " Oct 30 19:47:59.944561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.944533 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume" pod "19088b0a-79fa-4237-af46-4ed8847e00df" (UID: "19088b0a-79fa-4237-af46-4ed8847e00df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:59.944643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.944543 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\") pod \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\" (UID: \"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4\") " Oct 30 19:47:59.944946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.944919 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume" pod "bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" (UID: "bf5d0de9-6995-4b3f-ac55-7b3c42a660a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:59.945900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.945873 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19088b0a-79fa-4237-af46-4ed8847e00df-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:59.946003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.945906 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:47:59.946499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.946472 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm" pod "19088b0a-79fa-4237-af46-4ed8847e00df" (UID: "19088b0a-79fa-4237-af46-4ed8847e00df"). InnerVolumeSpecName "kube-api-access-5wljm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:59.947662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:47:59.947639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs" pod "bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" (UID: "bf5d0de9-6995-4b3f-ac55-7b3c42a660a4"). InnerVolumeSpecName "kube-api-access-2mvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:00.047217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.047182 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wljm\" (UniqueName: \"kubernetes.io/projected/19088b0a-79fa-4237-af46-4ed8847e00df-kube-api-access-5wljm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:00.047217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.047213 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvxs\" (UniqueName: \"kubernetes.io/projected/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4-kube-api-access-2mvxs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:00.459064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.459033 4990 generic.go:356] "Generic (PLEG): container finished" podID="19088b0a-79fa-4237-af46-4ed8847e00df" containerID="2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc" exitCode=0 Oct 30 19:48:00.459630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.459070 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" Oct 30 19:48:00.459630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.459126 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" event={"ID":"19088b0a-79fa-4237-af46-4ed8847e00df","Type":"ContainerDied","Data":"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc"} Oct 30 19:48:00.459630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.459176 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj" event={"ID":"19088b0a-79fa-4237-af46-4ed8847e00df","Type":"ContainerDied","Data":"1bc4abb3afb60d43e8afb4ce357da212a59b9dc4cf8359682075b10c41c061ed"} Oct 30 19:48:00.459630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.459204 4990 scope.go:122] "RemoveContainer" containerID="2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc" Oct 30 19:48:00.461370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.461332 4990 generic.go:356] "Generic (PLEG): container finished" podID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerID="c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce" exitCode=0 Oct 30 19:48:00.461370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.461362 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" Oct 30 19:48:00.461618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.461523 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" event={"ID":"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4","Type":"ContainerDied","Data":"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce"} Oct 30 19:48:00.461618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.461572 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf" event={"ID":"bf5d0de9-6995-4b3f-ac55-7b3c42a660a4","Type":"ContainerDied","Data":"73ec5fa5b7ffe0b1b1bf5fd8ff45bc4986131da329cbc69e1af2b7122d5f86a0"} Oct 30 19:48:00.473465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.473442 4990 scope.go:122] "RemoveContainer" containerID="2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc" Oct 30 19:48:00.473808 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:00.473783 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc\": container with ID starting with 2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc not found: ID does not exist" containerID="2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc" Oct 30 19:48:00.473882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.473818 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc"} err="failed to get container status \"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc\": rpc error: code = NotFound desc = could not find container \"2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc\": container with ID starting with 2a0b80ea28b1b35e2f26e53f08240fa028958e91950846ba62db6dd4896d2dcc not found: ID does not exist" Oct 30 19:48:00.473882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.473838 4990 scope.go:122] "RemoveContainer" containerID="c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce" Oct 30 19:48:00.484307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.484282 4990 scope.go:122] "RemoveContainer" containerID="c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce" Oct 30 19:48:00.484632 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:00.484611 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce\": container with ID starting with c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce not found: ID does not exist" containerID="c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce" Oct 30 19:48:00.484710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.484639 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce"} err="failed to get container status \"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce\": rpc error: code = NotFound desc = could not find container \"c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce\": container with ID starting with c4941a51a44fabeddf88a6d320f93c4d4288bd6a72c26e2daffafe39fc56f7ce not found: ID does not exist" Oct 30 19:48:00.509893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.509864 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:48:00.519273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.519242 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-614c02-6db95f5b6d-jkjxj"] Oct 30 19:48:00.538760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.538731 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:48:00.542717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.542692 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-614c02-5f58c5c8f9-jw9wf"] Oct 30 19:48:00.805335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.805296 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:00.977767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:00.977728 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:01.283172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:01.277544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:01.429288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:01.429200 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.405201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:02.405164 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" path="/var/lib/kubelet/pods/19088b0a-79fa-4237-af46-4ed8847e00df/volumes" Oct 30 19:48:02.405604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:02.405595 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" path="/var/lib/kubelet/pods/bf5d0de9-6995-4b3f-ac55-7b3c42a660a4/volumes" Oct 30 19:48:02.824625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:02.824585 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:04.510823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:04.510780 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:05.098584 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:05.098541 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:48:06.004113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.003773 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005431 4990 cpu_manager.go:414] "containerMap: removing container" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005460 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005502 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005510 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005887 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="19088b0a-79fa-4237-af46-4ed8847e00df" containerName="payload-processing" Oct 30 19:48:06.012114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.005918 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5d0de9-6995-4b3f-ac55-7b3c42a660a4" containerName="payload-pre-processing" Oct 30 19:48:06.020302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.020258 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.023361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.022370 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:48:06.034438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.034416 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"openshift-service-ca.crt\"" Oct 30 19:48:06.044438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.044415 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"kube-root-ca.crt\"" Oct 30 19:48:06.044582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.044543 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"default-dockercfg-qxdsx\"" Oct 30 19:48:06.044662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.044612 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"sub-test-model-d9eb76-kserve-self-signed-certs\"" Oct 30 19:48:06.088073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.088033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:06.142337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142286 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.142542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142400 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.142542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142506 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.142650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142630 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsvj\" (UniqueName: \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.142699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142664 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.142699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.142686 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.240255 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244201 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244296 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244446 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244550 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsvj\" (UniqueName: \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244583 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.244609 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.247529 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.248109 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.248786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.248360 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.260112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.255774 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.273102 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.266700 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.273349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.272899 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsvj\" (UniqueName: \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\") pod \"sub-test-model-d9eb76-kserve-796db989dc-6f44x\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.324736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.324680 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:48:06.337029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.336995 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:06.659209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:06.659181 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:48:06.661634 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:48:06.661596 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88f54105_8a8b_4bd0_8644_9394532154f3.slice/crio-812f0aa78fb9f19339d11df137bc499ebed6415f3eff2a3117e4e710583d603b WatchSource:0}: Error finding container 812f0aa78fb9f19339d11df137bc499ebed6415f3eff2a3117e4e710583d603b: Status 404 returned error can't find the container with id 812f0aa78fb9f19339d11df137bc499ebed6415f3eff2a3117e4e710583d603b Oct 30 19:48:07.526700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:07.526658 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" event={"ID":"88f54105-8a8b-4bd0-8644-9394532154f3","Type":"ContainerStarted","Data":"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a"} Oct 30 19:48:07.527161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:07.526712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" event={"ID":"88f54105-8a8b-4bd0-8644-9394532154f3","Type":"ContainerStarted","Data":"812f0aa78fb9f19339d11df137bc499ebed6415f3eff2a3117e4e710583d603b"} Oct 30 19:48:07.566152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:07.566080 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" podStartSLOduration=2.5660650499999997 podStartE2EDuration="2.56606505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:07.562711668 +0000 UTC m=+2595.274237034" watchObservedRunningTime="2026-10-30 19:48:07.56606505 +0000 UTC m=+2595.277590415" Oct 30 19:48:07.808171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:07.808129 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:08.337825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:08.337793 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:09.647650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:09.647587 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.295377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:11.295184 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:12.803871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:12.803833 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:48:12.804361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:12.804085 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" podUID="63c9d084-1785-49c7-8cd9-8dd74e40c639" containerName="maas-api" containerID="cri-o://03435b30a074153049136c59868c924e62f6517b284479041654759403822821" gracePeriod=30 Oct 30 19:48:12.913539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:12.913499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:13.161443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.161417 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:48:13.266337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.266293 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\") pod \"63c9d084-1785-49c7-8cd9-8dd74e40c639\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " Oct 30 19:48:13.266544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.266366 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\" (UniqueName: \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\") pod \"63c9d084-1785-49c7-8cd9-8dd74e40c639\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " Oct 30 19:48:13.266544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.266457 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\") pod \"63c9d084-1785-49c7-8cd9-8dd74e40c639\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " Oct 30 19:48:13.266670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.266653 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\") pod \"63c9d084-1785-49c7-8cd9-8dd74e40c639\" (UID: \"63c9d084-1785-49c7-8cd9-8dd74e40c639\") " Oct 30 19:48:13.268077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.268045 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca" pod "63c9d084-1785-49c7-8cd9-8dd74e40c639" (UID: "63c9d084-1785-49c7-8cd9-8dd74e40c639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:13.269149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.269035 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls" pod "63c9d084-1785-49c7-8cd9-8dd74e40c639" (UID: "63c9d084-1785-49c7-8cd9-8dd74e40c639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:13.270301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.270257 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs" pod "63c9d084-1785-49c7-8cd9-8dd74e40c639" (UID: "63c9d084-1785-49c7-8cd9-8dd74e40c639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:13.270485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.270458 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb" pod "63c9d084-1785-49c7-8cd9-8dd74e40c639" (UID: "63c9d084-1785-49c7-8cd9-8dd74e40c639"). InnerVolumeSpecName "kube-api-access-dv4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:13.380369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.380325 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:13.380369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.380374 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c9d084-1785-49c7-8cd9-8dd74e40c639-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:13.380659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.380393 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4fb\" (UniqueName: \"kubernetes.io/projected/63c9d084-1785-49c7-8cd9-8dd74e40c639-kube-api-access-dv4fb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:13.380659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.380410 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c9d084-1785-49c7-8cd9-8dd74e40c639-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:13.483530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.483445 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:13.576535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.576498 4990 generic.go:356] "Generic (PLEG): container finished" podID="63c9d084-1785-49c7-8cd9-8dd74e40c639" containerID="03435b30a074153049136c59868c924e62f6517b284479041654759403822821" exitCode=0 Oct 30 19:48:13.576849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.576535 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" event={"ID":"63c9d084-1785-49c7-8cd9-8dd74e40c639","Type":"ContainerDied","Data":"03435b30a074153049136c59868c924e62f6517b284479041654759403822821"} Oct 30 19:48:13.576849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.576604 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" event={"ID":"63c9d084-1785-49c7-8cd9-8dd74e40c639","Type":"ContainerDied","Data":"a9bd7ab7f23cea22d76f755347b256cdfa51b85f2cca9fcfcf1ec3fd0a011b1e"} Oct 30 19:48:13.576849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.576621 4990 scope.go:122] "RemoveContainer" containerID="03435b30a074153049136c59868c924e62f6517b284479041654759403822821" Oct 30 19:48:13.576849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.576604 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst" Oct 30 19:48:13.591629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.591604 4990 scope.go:122] "RemoveContainer" containerID="03435b30a074153049136c59868c924e62f6517b284479041654759403822821" Oct 30 19:48:13.591979 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:13.591950 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03435b30a074153049136c59868c924e62f6517b284479041654759403822821\": container with ID starting with 03435b30a074153049136c59868c924e62f6517b284479041654759403822821 not found: ID does not exist" containerID="03435b30a074153049136c59868c924e62f6517b284479041654759403822821" Oct 30 19:48:13.592047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.591986 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03435b30a074153049136c59868c924e62f6517b284479041654759403822821"} err="failed to get container status \"03435b30a074153049136c59868c924e62f6517b284479041654759403822821\": rpc error: code = NotFound desc = could not find container \"03435b30a074153049136c59868c924e62f6517b284479041654759403822821\": container with ID starting with 03435b30a074153049136c59868c924e62f6517b284479041654759403822821 not found: ID does not exist" Oct 30 19:48:13.631489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.631457 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:48:13.635649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.635619 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-09feeb-f8c5c79b7-k6lst"] Oct 30 19:48:13.639628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.639595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:13.796938 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:13.796836 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.146041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.146004 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.277700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.277666 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2"] Oct 30 19:48:14.278578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.278552 4990 cpu_manager.go:414] "containerMap: removing container" podUID="63c9d084-1785-49c7-8cd9-8dd74e40c639" containerName="maas-api" Oct 30 19:48:14.278578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.278576 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c9d084-1785-49c7-8cd9-8dd74e40c639" containerName="maas-api" Oct 30 19:48:14.278746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.278695 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c9d084-1785-49c7-8cd9-8dd74e40c639" containerName="maas-api" Oct 30 19:48:14.283552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.283534 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.291005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.290974 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.291308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.291266 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.291446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.291413 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmvw\" (UniqueName: \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.301524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.301491 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2"] Oct 30 19:48:14.340881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.340817 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:14.341083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.340984 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:48:14.392470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.392438 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.392637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.392589 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.392689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.392644 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmvw\" (UniqueName: \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.393452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.393422 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.395406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.395379 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.405481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.405421 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c9d084-1785-49c7-8cd9-8dd74e40c639" path="/var/lib/kubelet/pods/63c9d084-1785-49c7-8cd9-8dd74e40c639/volumes" Oct 30 19:48:14.410468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.410432 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmvw\" (UniqueName: \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\") pod \"maas-api-revoke-sub-498ea864-7rmb2\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.599582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.599543 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:14.912637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:14.912611 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2"] Oct 30 19:48:14.916169 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:48:14.916055 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25160e8_56c3_4ca9_b6c1_bb5311e134b8.slice/crio-0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19 WatchSource:0}: Error finding container 0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19: Status 404 returned error can't find the container with id 0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19 Oct 30 19:48:15.598972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:15.598927 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" event={"ID":"f25160e8-56c3-4ca9-b6c1-bb5311e134b8","Type":"ContainerStarted","Data":"cf21b7f3dded6f4008f5285072580f0b0ddae22c49a51b10fbcb90ff7fd668bc"} Oct 30 19:48:15.599465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:15.598995 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" event={"ID":"f25160e8-56c3-4ca9-b6c1-bb5311e134b8","Type":"ContainerStarted","Data":"0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19"} Oct 30 19:48:15.633992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:15.633942 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" podStartSLOduration=1.633928952 podStartE2EDuration="1.633928952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:15.632915914 +0000 UTC m=+2603.344441261" watchObservedRunningTime="2026-10-30 19:48:15.633928952 +0000 UTC m=+2603.345454318" Oct 30 19:48:15.727433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:15.727389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:16.324742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.324703 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:48:16.338060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.338030 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:16.350760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.350732 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:16.609401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.609281 4990 generic.go:356] "Generic (PLEG): container finished" podID="f25160e8-56c3-4ca9-b6c1-bb5311e134b8" containerID="cf21b7f3dded6f4008f5285072580f0b0ddae22c49a51b10fbcb90ff7fd668bc" exitCode=0 Oct 30 19:48:16.609401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.609381 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" event={"ID":"f25160e8-56c3-4ca9-b6c1-bb5311e134b8","Type":"ContainerDied","Data":"cf21b7f3dded6f4008f5285072580f0b0ddae22c49a51b10fbcb90ff7fd668bc"} Oct 30 19:48:16.621742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.621713 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:48:16.882217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.882185 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:48:16.928738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.928562 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\" (UniqueName: \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\") pod \"3afe5982-06f8-4993-99f9-8e268b1397f0\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " Oct 30 19:48:16.929058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.929023 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\") pod \"3afe5982-06f8-4993-99f9-8e268b1397f0\" (UID: \"3afe5982-06f8-4993-99f9-8e268b1397f0\") " Oct 30 19:48:16.930216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.930170 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume" pod "3afe5982-06f8-4993-99f9-8e268b1397f0" (UID: "3afe5982-06f8-4993-99f9-8e268b1397f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:16.932731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:16.932706 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2" pod "3afe5982-06f8-4993-99f9-8e268b1397f0" (UID: "3afe5982-06f8-4993-99f9-8e268b1397f0"). InnerVolumeSpecName "kube-api-access-r46l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:17.031129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.031068 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46l2\" (UniqueName: \"kubernetes.io/projected/3afe5982-06f8-4993-99f9-8e268b1397f0-kube-api-access-r46l2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:17.031129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.031123 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3afe5982-06f8-4993-99f9-8e268b1397f0-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:17.492118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.491651 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:17.619709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.619667 4990 generic.go:356] "Generic (PLEG): container finished" podID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerID="3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912" exitCode=0 Oct 30 19:48:17.620254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.619723 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" Oct 30 19:48:17.620254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.619746 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" event={"ID":"3afe5982-06f8-4993-99f9-8e268b1397f0","Type":"ContainerDied","Data":"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912"} Oct 30 19:48:17.620254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.619803 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb" event={"ID":"3afe5982-06f8-4993-99f9-8e268b1397f0","Type":"ContainerDied","Data":"057d2507a0fe2986070ac26cdb347cffd4dbf65d42ccca9eca83689c36c9574e"} Oct 30 19:48:17.620254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.619825 4990 scope.go:122] "RemoveContainer" containerID="3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912" Oct 30 19:48:17.633212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.633192 4990 scope.go:122] "RemoveContainer" containerID="3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912" Oct 30 19:48:17.633500 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:17.633478 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912\": container with ID starting with 3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912 not found: ID does not exist" containerID="3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912" Oct 30 19:48:17.633572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.633511 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912"} err="failed to get container status \"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912\": rpc error: code = NotFound desc = could not find container \"3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912\": container with ID starting with 3a629d33ae95c80183593e79b761615fde72f50918e3c95e0682ec25561d2912 not found: ID does not exist" Oct 30 19:48:17.667418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.666940 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:48:17.675239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.675211 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09feeb-787c67455f-rg4mb"] Oct 30 19:48:17.801591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.801562 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:17.840830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.840796 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\") pod \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " Oct 30 19:48:17.841074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.841009 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\") pod \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " Oct 30 19:48:17.841074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.841063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\" (UniqueName: \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\") pod \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\" (UID: \"f25160e8-56c3-4ca9-b6c1-bb5311e134b8\") " Oct 30 19:48:17.841660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.841626 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca" pod "f25160e8-56c3-4ca9-b6c1-bb5311e134b8" (UID: "f25160e8-56c3-4ca9-b6c1-bb5311e134b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:17.841894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.841875 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:17.843546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.843516 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw" pod "f25160e8-56c3-4ca9-b6c1-bb5311e134b8" (UID: "f25160e8-56c3-4ca9-b6c1-bb5311e134b8"). InnerVolumeSpecName "kube-api-access-ddmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:17.843706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.843601 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca" pod "f25160e8-56c3-4ca9-b6c1-bb5311e134b8" (UID: "f25160e8-56c3-4ca9-b6c1-bb5311e134b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:17.943050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.943016 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:17.943050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:17.943046 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddmvw\" (UniqueName: \"kubernetes.io/projected/f25160e8-56c3-4ca9-b6c1-bb5311e134b8-kube-api-access-ddmvw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:18.408996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:18.408951 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" path="/var/lib/kubelet/pods/3afe5982-06f8-4993-99f9-8e268b1397f0/volumes" Oct 30 19:48:18.630966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:18.630935 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" Oct 30 19:48:18.631344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:18.630974 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2" event={"ID":"f25160e8-56c3-4ca9-b6c1-bb5311e134b8","Type":"ContainerDied","Data":"0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19"} Oct 30 19:48:18.631344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:18.631023 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0172fcd6198bd32020c7e5d05a91b67d9c907fda1c41902bd132ebd5d7104f19" Oct 30 19:48:19.366460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:19.361465 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.459932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:20.459895 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.706158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:20.706125 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.936408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:20.936363 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.397502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:21.397458 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.723576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:21.723495 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.159329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.159283 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:48:22.188477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.188443 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:48:22.193246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.193213 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.353654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.353619 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.513228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.512907 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.931444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.931402 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:48:22.933177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933148 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f25160e8-56c3-4ca9-b6c1-bb5311e134b8" containerName="revoke-keys" Oct 30 19:48:22.933346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933332 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25160e8-56c3-4ca9-b6c1-bb5311e134b8" containerName="revoke-keys" Oct 30 19:48:22.933499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933487 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" Oct 30 19:48:22.933605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933593 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" Oct 30 19:48:22.933851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933839 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25160e8-56c3-4ca9-b6c1-bb5311e134b8" containerName="revoke-keys" Oct 30 19:48:22.933955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.933946 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afe5982-06f8-4993-99f9-8e268b1397f0" containerName="payload-pre-processing" Oct 30 19:48:22.941667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.939719 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:22.951947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.951920 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-54086b-serving-cert\"" Oct 30 19:48:22.973219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:22.973187 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:48:23.024567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.024522 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh26h\" (UniqueName: \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.024780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.024739 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.024864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.024781 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.024945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.024927 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.071183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.071143 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.125817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.125779 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.126003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.125896 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh26h\" (UniqueName: \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.126131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.126045 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.126131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.126068 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.126787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.126764 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.130166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.130138 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.132364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.132331 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.148475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.148447 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh26h\" (UniqueName: \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\") pod \"maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.223650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.223562 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.271111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.271059 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:23.368637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.368497 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.608117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.608072 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:48:23.609990 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:48:23.609954 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51006b93_2179_4679_a433_fedbac4e8f16.slice/crio-19383aec75303d1703b931b54571db9fabd6472118d657a45a7aea9fb14ad00c WatchSource:0}: Error finding container 19383aec75303d1703b931b54571db9fabd6472118d657a45a7aea9fb14ad00c: Status 404 returned error can't find the container with id 19383aec75303d1703b931b54571db9fabd6472118d657a45a7aea9fb14ad00c Oct 30 19:48:23.611841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.611818 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:48:23.669880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:23.669840 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" event={"ID":"51006b93-2179-4679-a433-fedbac4e8f16","Type":"ContainerStarted","Data":"19383aec75303d1703b931b54571db9fabd6472118d657a45a7aea9fb14ad00c"} Oct 30 19:48:24.681730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:24.681687 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" event={"ID":"51006b93-2179-4679-a433-fedbac4e8f16","Type":"ContainerStarted","Data":"fd993f1e551098c18fb6ef7273a764785de87b6e36bc334f6cc7c5fb64f46cba"} Oct 30 19:48:24.682194 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:24.681760 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:24.722620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:24.722560 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" podStartSLOduration=2.390078864 podStartE2EDuration="2.722542985s" totalImagesPullingTime="332.464121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:24.718669317 +0000 UTC m=+2612.430194695" watchObservedRunningTime="2026-10-30 19:48:24.722542985 +0000 UTC m=+2612.434068351" Oct 30 19:48:24.899533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:24.899448 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:25.257612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:25.257525 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.259130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:26.259063 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.414517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:26.414480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.991429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:26.991393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:27.159821 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.159763 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" podUID="b9254072-5a4b-4007-bc98-8e0b42946644" containerName="payload-processing" containerID="cri-o://5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722" gracePeriod=25 Oct 30 19:48:27.189740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.189698 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" containerID="cri-o://12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47" gracePeriod=25 Oct 30 19:48:27.446880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.446853 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:48:27.498927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.498891 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\" (UniqueName: \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\") pod \"b9254072-5a4b-4007-bc98-8e0b42946644\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " Oct 30 19:48:27.499142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.498947 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\") pod \"b9254072-5a4b-4007-bc98-8e0b42946644\" (UID: \"b9254072-5a4b-4007-bc98-8e0b42946644\") " Oct 30 19:48:27.499557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.499469 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume" pod "b9254072-5a4b-4007-bc98-8e0b42946644" (UID: "b9254072-5a4b-4007-bc98-8e0b42946644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:27.500046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.500020 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9254072-5a4b-4007-bc98-8e0b42946644-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:27.502310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.502282 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv" pod "b9254072-5a4b-4007-bc98-8e0b42946644" (UID: "b9254072-5a4b-4007-bc98-8e0b42946644"). InnerVolumeSpecName "kube-api-access-zqlxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:27.601989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.601944 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlxv\" (UniqueName: \"kubernetes.io/projected/b9254072-5a4b-4007-bc98-8e0b42946644-kube-api-access-zqlxv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:27.713711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.713671 4990 generic.go:356] "Generic (PLEG): container finished" podID="b9254072-5a4b-4007-bc98-8e0b42946644" containerID="5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722" exitCode=0 Oct 30 19:48:27.713711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.713716 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" Oct 30 19:48:27.713978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.713719 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" event={"ID":"b9254072-5a4b-4007-bc98-8e0b42946644","Type":"ContainerDied","Data":"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722"} Oct 30 19:48:27.713978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.713834 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj" event={"ID":"b9254072-5a4b-4007-bc98-8e0b42946644","Type":"ContainerDied","Data":"7390f8af094b45a5e231795b29eaa2b70ae9be5c3cdd6ebfcb7f500e8db0fc35"} Oct 30 19:48:27.713978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.713865 4990 scope.go:122] "RemoveContainer" containerID="5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722" Oct 30 19:48:27.724856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.724838 4990 scope.go:122] "RemoveContainer" containerID="5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722" Oct 30 19:48:27.725126 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:27.725102 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722\": container with ID starting with 5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722 not found: ID does not exist" containerID="5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722" Oct 30 19:48:27.725183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.725134 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722"} err="failed to get container status \"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722\": rpc error: code = NotFound desc = could not find container \"5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722\": container with ID starting with 5a890c6820dbf82b27ff5a798b7c1b65702cb0108a7faf68be4216c298654722 not found: ID does not exist" Oct 30 19:48:27.760015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.759958 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:48:27.768780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.768752 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3504b-6d6d876cb4-nrvhj"] Oct 30 19:48:27.917673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:27.917576 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:28.029166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.029131 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 19:48:28.029814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.029801 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b9254072-5a4b-4007-bc98-8e0b42946644" containerName="payload-processing" Oct 30 19:48:28.029865 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.029815 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9254072-5a4b-4007-bc98-8e0b42946644" containerName="payload-processing" Oct 30 19:48:28.029972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.029960 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9254072-5a4b-4007-bc98-8e0b42946644" containerName="payload-processing" Oct 30 19:48:28.034977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.034958 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.050157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.050133 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-ec2fe7\"" Oct 30 19:48:28.054150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.054070 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 19:48:28.065799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.065771 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:28.119791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.119597 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnm6p\" (UniqueName: \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.119791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.119675 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.158113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.157427 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-ec2fe7-dockercfg-44xz2\"" Oct 30 19:48:28.224162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.223272 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm6p\" (UniqueName: \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.232115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.229232 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.232115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.230574 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.269121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.238138 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 19:48:28.271736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.270874 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.299769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.299723 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm6p\" (UniqueName: \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\") pod \"payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.308268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.308237 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 19:48:28.331462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.331434 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl76j\" (UniqueName: \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.331598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.331476 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.349602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.349572 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:28.406392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.406356 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9254072-5a4b-4007-bc98-8e0b42946644" path="/var/lib/kubelet/pods/b9254072-5a4b-4007-bc98-8e0b42946644/volumes" Oct 30 19:48:28.434290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.433880 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl76j\" (UniqueName: \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.434290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.433938 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.434807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.434785 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.450825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.450802 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl76j\" (UniqueName: \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\") pod \"payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.627709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.627682 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:28.661698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.661671 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 19:48:28.663624 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:48:28.663592 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6577b72_e951_4092_a434_648c4dec7b60.slice/crio-ff4d1d287ee36e78263d99e0d90391070225acab4c1cadcaa0d64adaa70c3341 WatchSource:0}: Error finding container ff4d1d287ee36e78263d99e0d90391070225acab4c1cadcaa0d64adaa70c3341: Status 404 returned error can't find the container with id ff4d1d287ee36e78263d99e0d90391070225acab4c1cadcaa0d64adaa70c3341 Oct 30 19:48:28.730041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.729998 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" event={"ID":"a6577b72-e951-4092-a434-648c4dec7b60","Type":"ContainerStarted","Data":"ff4d1d287ee36e78263d99e0d90391070225acab4c1cadcaa0d64adaa70c3341"} Oct 30 19:48:28.850128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.849446 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:28.950112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:28.950069 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 19:48:28.951781 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:48:28.951753 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae0d1d6_1cb9_4f69_ac9a_f6b72cd831fa.slice/crio-24d88508159ad7aaf9136b1a040ed462107ce2be8f4f824619a431ad42f3ba74 WatchSource:0}: Error finding container 24d88508159ad7aaf9136b1a040ed462107ce2be8f4f824619a431ad42f3ba74: Status 404 returned error can't find the container with id 24d88508159ad7aaf9136b1a040ed462107ce2be8f4f824619a431ad42f3ba74 Oct 30 19:48:29.742643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.742608 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" event={"ID":"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa","Type":"ContainerStarted","Data":"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78"} Oct 30 19:48:29.743064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.742659 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" event={"ID":"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa","Type":"ContainerStarted","Data":"24d88508159ad7aaf9136b1a040ed462107ce2be8f4f824619a431ad42f3ba74"} Oct 30 19:48:29.743064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.742712 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:29.744522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.744492 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" event={"ID":"a6577b72-e951-4092-a434-648c4dec7b60","Type":"ContainerStarted","Data":"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61"} Oct 30 19:48:29.744654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.744547 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:29.789959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.789895 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" podStartSLOduration=1.789876391 podStartE2EDuration="1.789876391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:29.776001526 +0000 UTC m=+2617.487526893" watchObservedRunningTime="2026-10-30 19:48:29.789876391 +0000 UTC m=+2617.501401755" Oct 30 19:48:29.810267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:29.810211 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" podStartSLOduration=1.8101933730000002 podStartE2EDuration="1.810193373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:29.810068831 +0000 UTC m=+2617.521594197" watchObservedRunningTime="2026-10-30 19:48:29.810193373 +0000 UTC m=+2617.521718739" Oct 30 19:48:30.419803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:30.419479 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:30.697949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:30.697864 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:48:32.017831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:32.017791 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:33.805639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:33.805596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:35.099712 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:35.099672 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:48:35.421763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:35.421668 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:36.816803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:36.816762 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:48:37.155558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:37.155443 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.030728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:39.030692 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.599451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:40.599402 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.756364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:40.756335 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 19:48:40.756538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:40.756441 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 19:48:42.303458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:42.303373 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:43.997874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:43.997834 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:45.289475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:45.289424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:45.945492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:45.945460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.095815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:46.095776 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.239973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:46.239877 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.815998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:46.815952 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:48:47.466599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.466535 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:48:47.605856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.605823 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\" (UniqueName: \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\") pod \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " Oct 30 19:48:47.606054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.605945 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\") pod \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\" (UID: \"2d45b28d-aea3-4df1-98b9-64de5947a3ad\") " Oct 30 19:48:47.606357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.606314 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume" pod "2d45b28d-aea3-4df1-98b9-64de5947a3ad" (UID: "2d45b28d-aea3-4df1-98b9-64de5947a3ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:47.606513 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.606496 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d45b28d-aea3-4df1-98b9-64de5947a3ad-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:47.608200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.608171 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26" pod "2d45b28d-aea3-4df1-98b9-64de5947a3ad" (UID: "2d45b28d-aea3-4df1-98b9-64de5947a3ad"). InnerVolumeSpecName "kube-api-access-cbt26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:47.708459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.708423 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbt26\" (UniqueName: \"kubernetes.io/projected/2d45b28d-aea3-4df1-98b9-64de5947a3ad-kube-api-access-cbt26\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:48:47.908777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.908741 4990 generic.go:356] "Generic (PLEG): container finished" podID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerID="12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47" exitCode=0 Oct 30 19:48:47.909258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.908799 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" Oct 30 19:48:47.909258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.908825 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" event={"ID":"2d45b28d-aea3-4df1-98b9-64de5947a3ad","Type":"ContainerDied","Data":"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47"} Oct 30 19:48:47.909258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.908893 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd" event={"ID":"2d45b28d-aea3-4df1-98b9-64de5947a3ad","Type":"ContainerDied","Data":"19d4e4eb3198f4ab5de076cbfa19e63c611d30a106e93e0fd3658a8d9c11eb93"} Oct 30 19:48:47.909258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.908920 4990 scope.go:122] "RemoveContainer" containerID="12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47" Oct 30 19:48:47.922325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.922293 4990 scope.go:122] "RemoveContainer" containerID="12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47" Oct 30 19:48:47.922656 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:48:47.922630 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47\": container with ID starting with 12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47 not found: ID does not exist" containerID="12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47" Oct 30 19:48:47.922762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.922666 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47"} err="failed to get container status \"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47\": rpc error: code = NotFound desc = could not find container \"12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47\": container with ID starting with 12834aa2bf32d11c582726038ed345dfadbec8785a097d63e8c562b0e0206c47 not found: ID does not exist" Oct 30 19:48:47.958303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.958281 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:48:47.968561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:47.968533 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3504b-78b894fbc9-pjjtd"] Oct 30 19:48:48.030176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:48.030141 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.403173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:48.403142 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" path="/var/lib/kubelet/pods/2d45b28d-aea3-4df1-98b9-64de5947a3ad/volumes" Oct 30 19:48:49.635930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:49.635222 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:51.431818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:51.431778 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:52.170346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:52.170306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:53.419930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:53.419196 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:54.998779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:54.998741 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:56.225914 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:56.225877 4990 scope.go:122] "RemoveContainer" containerID="4b4849dfe8781ffaf3f7e902338e36d8369adfdc25b3330cfe045699bc08f2f1" Oct 30 19:48:56.577412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:56.577379 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:56.743241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:56.743191 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:58.357037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:48:58.356996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:00.003292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:00.003253 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.430113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:01.428948 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.762107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:01.762004 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.906575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:01.906537 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.201243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:03.201201 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.350287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:03.350254 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.505331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:03.505233 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.736237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:03.736204 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.898987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:03.898952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:04.043306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:04.043266 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:04.523570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:04.523533 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:05.096600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.093764 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:05.141285 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:05.141246 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:49:05.292208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.292169 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:49:05.292737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.292694 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" podUID="25cc5e20-2452-4799-937e-bd2c0c7a339c" containerName="maas-api" containerID="cri-o://d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c" gracePeriod=30 Oct 30 19:49:05.346309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.346246 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:05.528499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.528324 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:05.934979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.934946 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:49:05.996595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.996558 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\") pod \"25cc5e20-2452-4799-937e-bd2c0c7a339c\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " Oct 30 19:49:05.996768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.996611 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") pod \"25cc5e20-2452-4799-937e-bd2c0c7a339c\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " Oct 30 19:49:05.996768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.996643 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\" (UniqueName: \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\") pod \"25cc5e20-2452-4799-937e-bd2c0c7a339c\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " Oct 30 19:49:05.996768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.996740 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\") pod \"25cc5e20-2452-4799-937e-bd2c0c7a339c\" (UID: \"25cc5e20-2452-4799-937e-bd2c0c7a339c\") " Oct 30 19:49:05.996958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.996916 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca" pod "25cc5e20-2452-4799-937e-bd2c0c7a339c" (UID: "25cc5e20-2452-4799-937e-bd2c0c7a339c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:05.997481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.997454 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc5e20-2452-4799-937e-bd2c0c7a339c-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:05.999606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:05.999578 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs" pod "25cc5e20-2452-4799-937e-bd2c0c7a339c" (UID: "25cc5e20-2452-4799-937e-bd2c0c7a339c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:06.002344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.002305 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls" pod "25cc5e20-2452-4799-937e-bd2c0c7a339c" (UID: "25cc5e20-2452-4799-937e-bd2c0c7a339c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:06.018740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.018710 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575" pod "25cc5e20-2452-4799-937e-bd2c0c7a339c" (UID: "25cc5e20-2452-4799-937e-bd2c0c7a339c"). InnerVolumeSpecName "kube-api-access-jw575". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:06.024873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.024788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.094874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.094842 4990 generic.go:356] "Generic (PLEG): container finished" podID="25cc5e20-2452-4799-937e-bd2c0c7a339c" containerID="d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c" exitCode=0 Oct 30 19:49:06.095001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.094884 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" Oct 30 19:49:06.095001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.094920 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" event={"ID":"25cc5e20-2452-4799-937e-bd2c0c7a339c","Type":"ContainerDied","Data":"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c"} Oct 30 19:49:06.095001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.094972 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59" event={"ID":"25cc5e20-2452-4799-937e-bd2c0c7a339c","Type":"ContainerDied","Data":"16f530d5105ed22847246bb4790feffd7ea6c13541cbf7c62ecfadfca41237fd"} Oct 30 19:49:06.095001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.094998 4990 scope.go:122] "RemoveContainer" containerID="d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c" Oct 30 19:49:06.098344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.098322 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:06.098344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.098345 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw575\" (UniqueName: \"kubernetes.io/projected/25cc5e20-2452-4799-937e-bd2c0c7a339c-kube-api-access-jw575\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:06.098518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.098355 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25cc5e20-2452-4799-937e-bd2c0c7a339c-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:06.106572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.106553 4990 scope.go:122] "RemoveContainer" containerID="d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c" Oct 30 19:49:06.106832 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:06.106813 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c\": container with ID starting with d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c not found: ID does not exist" containerID="d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c" Oct 30 19:49:06.106900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.106840 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c"} err="failed to get container status \"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c\": rpc error: code = NotFound desc = could not find container \"d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c\": container with ID starting with d58397626d336603845aef1dc9aadbd73b8287235ec40935ed88ddd0af05b54c not found: ID does not exist" Oct 30 19:49:06.144282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.144235 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:49:06.148650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.148627 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f6cef-68b67dc766-nmj59"] Oct 30 19:49:06.182631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.182557 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.402488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.402400 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cc5e20-2452-4799-937e-bd2c0c7a339c" path="/var/lib/kubelet/pods/25cc5e20-2452-4799-937e-bd2c0c7a339c/volumes" Oct 30 19:49:06.745226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:06.745139 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:08.092118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:08.091060 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:09.722511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:09.722423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.439983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:11.432743 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.799873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:12.799823 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.937383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:12.937345 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.085406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:13.085289 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.232981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:13.232942 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:14.596113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:14.586750 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:16.185662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.182358 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:16.292574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.292535 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:49:16.323904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.323855 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:49:16.470547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.470458 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:16.616224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.616190 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:16.764306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:16.764205 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.121305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:18.121260 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.789898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:18.789859 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.940240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:18.940174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:20.148187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:20.148145 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:20.606512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:20.606479 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:20.777890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:20.777828 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:21.293515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.293469 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" podUID="dedadb80-d646-4b09-abd3-23cdafab6bf0" containerName="payload-processing" containerID="cri-o://532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390" gracePeriod=25 Oct 30 19:49:21.324562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.324523 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" podUID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" containerName="payload-pre-processing" containerID="cri-o://786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7" gracePeriod=25 Oct 30 19:49:21.657681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.657654 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:49:21.680502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.680474 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:49:21.725195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.725154 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\") pod \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " Oct 30 19:49:21.725411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.725325 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\" (UniqueName: \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\") pod \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\" (UID: \"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c\") " Oct 30 19:49:21.725596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.725564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume" pod "237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" (UID: "237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:21.726295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.726271 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:21.727907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.727883 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj" pod "237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" (UID: "237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c"). InnerVolumeSpecName "kube-api-access-6vxsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.833983 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\" (UniqueName: \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\") pod \"dedadb80-d646-4b09-abd3-23cdafab6bf0\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.834057 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\") pod \"dedadb80-d646-4b09-abd3-23cdafab6bf0\" (UID: \"dedadb80-d646-4b09-abd3-23cdafab6bf0\") " Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.842686 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxsj\" (UniqueName: \"kubernetes.io/projected/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c-kube-api-access-6vxsj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.854602 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume" pod "dedadb80-d646-4b09-abd3-23cdafab6bf0" (UID: "dedadb80-d646-4b09-abd3-23cdafab6bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.854754 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh" pod "dedadb80-d646-4b09-abd3-23cdafab6bf0" (UID: "dedadb80-d646-4b09-abd3-23cdafab6bf0"). InnerVolumeSpecName "kube-api-access-qnkhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:21.862169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.855868 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:21.944626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.944587 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnkhh\" (UniqueName: \"kubernetes.io/projected/dedadb80-d646-4b09-abd3-23cdafab6bf0-kube-api-access-qnkhh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:21.944626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:21.944620 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedadb80-d646-4b09-abd3-23cdafab6bf0-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:22.278783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.278670 4990 generic.go:356] "Generic (PLEG): container finished" podID="dedadb80-d646-4b09-abd3-23cdafab6bf0" containerID="532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390" exitCode=0 Oct 30 19:49:22.278783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.278739 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" Oct 30 19:49:22.278989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.278840 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" event={"ID":"dedadb80-d646-4b09-abd3-23cdafab6bf0","Type":"ContainerDied","Data":"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390"} Oct 30 19:49:22.278989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.278927 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj" event={"ID":"dedadb80-d646-4b09-abd3-23cdafab6bf0","Type":"ContainerDied","Data":"a493527b9a08d9a24f1e0e5d03ec545df81c6aed2e2fdeed635d7ab30359dd9f"} Oct 30 19:49:22.278989 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.278961 4990 scope.go:122] "RemoveContainer" containerID="532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390" Oct 30 19:49:22.286206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.286176 4990 generic.go:356] "Generic (PLEG): container finished" podID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" containerID="786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7" exitCode=0 Oct 30 19:49:22.286313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.286242 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" Oct 30 19:49:22.286395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.286377 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" event={"ID":"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c","Type":"ContainerDied","Data":"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7"} Oct 30 19:49:22.286463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.286429 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh" event={"ID":"237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c","Type":"ContainerDied","Data":"447406b9e3ff9a02b5561d6433e29b986f738a0c6fe8b3ecaef7239c4e81b082"} Oct 30 19:49:22.298615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.298419 4990 scope.go:122] "RemoveContainer" containerID="532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390" Oct 30 19:49:22.298929 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:22.298767 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390\": container with ID starting with 532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390 not found: ID does not exist" containerID="532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390" Oct 30 19:49:22.298929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.298805 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390"} err="failed to get container status \"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390\": rpc error: code = NotFound desc = could not find container \"532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390\": container with ID starting with 532483cd5e87363234b68a6cc1faaf06b805945bf8418cf918e00a60ced44390 not found: ID does not exist" Oct 30 19:49:22.298929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.298829 4990 scope.go:122] "RemoveContainer" containerID="786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7" Oct 30 19:49:22.321387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.321357 4990 scope.go:122] "RemoveContainer" containerID="786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7" Oct 30 19:49:22.321743 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:22.321717 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7\": container with ID starting with 786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7 not found: ID does not exist" containerID="786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7" Oct 30 19:49:22.321800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.321756 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7"} err="failed to get container status \"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7\": rpc error: code = NotFound desc = could not find container \"786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7\": container with ID starting with 786448e052096cf90ea552ac6e9420d07fa37d78339019e678aa3e9e4a4ff1d7 not found: ID does not exist" Oct 30 19:49:22.338469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.338438 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:49:22.347616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.347580 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-68dcf85c6b-5sdqj"] Oct 30 19:49:22.382410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.379635 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:49:22.383804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.383772 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-58fb858499-4jflh"] Oct 30 19:49:22.406726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.406683 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" path="/var/lib/kubelet/pods/237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c/volumes" Oct 30 19:49:22.407364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:22.407344 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedadb80-d646-4b09-abd3-23cdafab6bf0" path="/var/lib/kubelet/pods/dedadb80-d646-4b09-abd3-23cdafab6bf0/volumes" Oct 30 19:49:23.611324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:23.611284 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.199259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:26.199218 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.265260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:28.265214 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:29.017799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:29.017761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:30.115577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:30.115521 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:32.021393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:32.021347 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:33.662887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:33.662813 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.106890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:34.106848 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:49:34.286764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:34.286724 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.430879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:34.430788 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:35.099771 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:35.099726 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:49:35.334204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:35.333797 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:37.129436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:37.129399 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.060281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.060240 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.108008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.107970 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" podUID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" containerName="payload-processing" containerID="cri-o://5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278" gracePeriod=25 Oct 30 19:49:39.414671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.414635 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:49:39.476788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.476749 4990 generic.go:356] "Generic (PLEG): container finished" podID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" containerID="5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278" exitCode=0 Oct 30 19:49:39.476788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.476797 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" Oct 30 19:49:39.477079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.476833 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" event={"ID":"5e3e3980-81ef-40bc-9be0-ef75634ffa75","Type":"ContainerDied","Data":"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278"} Oct 30 19:49:39.477079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.476913 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj" event={"ID":"5e3e3980-81ef-40bc-9be0-ef75634ffa75","Type":"ContainerDied","Data":"b55c8b23d8079e21669e777b6b44aec2f3b6d6e3d85e7575c0a369eb0992bda9"} Oct 30 19:49:39.477079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.476944 4990 scope.go:122] "RemoveContainer" containerID="5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278" Oct 30 19:49:39.490431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.490407 4990 scope.go:122] "RemoveContainer" containerID="5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278" Oct 30 19:49:39.490834 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:49:39.490806 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278\": container with ID starting with 5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278 not found: ID does not exist" containerID="5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278" Oct 30 19:49:39.490958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.490840 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278"} err="failed to get container status \"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278\": rpc error: code = NotFound desc = could not find container \"5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278\": container with ID starting with 5dfefca4dbd04da6d5b96a4487515ae9cf8b06097ab43fe02d7ec6494c2ad278 not found: ID does not exist" Oct 30 19:49:39.540417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.540385 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\") pod \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " Oct 30 19:49:39.540620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.540485 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\" (UniqueName: \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\") pod \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\" (UID: \"5e3e3980-81ef-40bc-9be0-ef75634ffa75\") " Oct 30 19:49:39.540790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.540765 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume" pod "5e3e3980-81ef-40bc-9be0-ef75634ffa75" (UID: "5e3e3980-81ef-40bc-9be0-ef75634ffa75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:39.541270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.541244 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e3e3980-81ef-40bc-9be0-ef75634ffa75-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:39.542744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.542718 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v" pod "5e3e3980-81ef-40bc-9be0-ef75634ffa75" (UID: "5e3e3980-81ef-40bc-9be0-ef75634ffa75"). InnerVolumeSpecName "kube-api-access-4g57v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:39.642651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.642556 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g57v\" (UniqueName: \"kubernetes.io/projected/5e3e3980-81ef-40bc-9be0-ef75634ffa75-kube-api-access-4g57v\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:49:39.823346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.823310 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:49:39.831987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:39.831950 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-576a62-c5d8ffb64-z8dkj"] Oct 30 19:49:40.421907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.421858 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" path="/var/lib/kubelet/pods/5e3e3980-81ef-40bc-9be0-ef75634ffa75/volumes" Oct 30 19:49:40.481560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.481520 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.697704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.697612 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.965240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.965148 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:49:40.966578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.966547 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" Oct 30 19:49:40.966758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.966743 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" Oct 30 19:49:40.966873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.966862 4990 cpu_manager.go:414] "containerMap: removing container" podUID="dedadb80-d646-4b09-abd3-23cdafab6bf0" containerName="payload-processing" Oct 30 19:49:40.966955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.966945 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedadb80-d646-4b09-abd3-23cdafab6bf0" containerName="payload-processing" Oct 30 19:49:40.967035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967026 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" containerName="payload-processing" Oct 30 19:49:40.967130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967118 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" containerName="payload-processing" Oct 30 19:49:40.967249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967239 4990 cpu_manager.go:414] "containerMap: removing container" podUID="25cc5e20-2452-4799-937e-bd2c0c7a339c" containerName="maas-api" Oct 30 19:49:40.967319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967309 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cc5e20-2452-4799-937e-bd2c0c7a339c" containerName="maas-api" Oct 30 19:49:40.967398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967388 4990 cpu_manager.go:414] "containerMap: removing container" podUID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" containerName="payload-pre-processing" Oct 30 19:49:40.967469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967460 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" containerName="payload-pre-processing" Oct 30 19:49:40.989115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.967708 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cc5e20-2452-4799-937e-bd2c0c7a339c" containerName="maas-api" Oct 30 19:49:40.989404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.989375 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="237f02be-7d7d-4b1c-bc7d-e6f6f77f0c4c" containerName="payload-pre-processing" Oct 30 19:49:40.989538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.989524 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d45b28d-aea3-4df1-98b9-64de5947a3ad" containerName="payload-pre-processing" Oct 30 19:49:40.989657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.989645 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedadb80-d646-4b09-abd3-23cdafab6bf0" containerName="payload-processing" Oct 30 19:49:40.989744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:40.989734 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3e3980-81ef-40bc-9be0-ef75634ffa75" containerName="payload-processing" Oct 30 19:49:41.028004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.006960 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.055725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.055680 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:49:41.056547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.056444 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54086b\"" Oct 30 19:49:41.056708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.056604 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54086b-dockercfg-nldlt\"" Oct 30 19:49:41.098130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.098041 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.098580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.098553 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwjf\" (UniqueName: \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.200828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.200785 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.201019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.200932 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwjf\" (UniqueName: \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.201637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.201613 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.218551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.218473 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwjf\" (UniqueName: \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\") pod \"payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.381340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.381292 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:41.463222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.457754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:41.700166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.700135 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:49:41.702805 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:49:41.702776 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2e1510_0528_4d5f_a9e0_a9924e46206a.slice/crio-d185f4c4742ba385e609c3ff2b73e90c84b6ce5b4f0ff73d0b5edba8a1fcf90d WatchSource:0}: Error finding container d185f4c4742ba385e609c3ff2b73e90c84b6ce5b4f0ff73d0b5edba8a1fcf90d: Status 404 returned error can't find the container with id d185f4c4742ba385e609c3ff2b73e90c84b6ce5b4f0ff73d0b5edba8a1fcf90d Oct 30 19:49:41.855939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:41.855903 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.509239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:42.509143 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" event={"ID":"7b2e1510-0528-4d5f-a9e0-a9924e46206a","Type":"ContainerStarted","Data":"d185f4c4742ba385e609c3ff2b73e90c84b6ce5b4f0ff73d0b5edba8a1fcf90d"} Oct 30 19:49:43.519169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:43.519131 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" event={"ID":"7b2e1510-0528-4d5f-a9e0-a9924e46206a","Type":"ContainerStarted","Data":"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8"} Oct 30 19:49:43.519602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:43.519197 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:43.555908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:43.555562 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" podStartSLOduration=1.870891931 podStartE2EDuration="3.555542697s" totalImagesPullingTime="1.684650766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.553631838 +0000 UTC m=+2691.265157205" watchObservedRunningTime="2026-10-30 19:49:43.555542697 +0000 UTC m=+2691.267068064" Oct 30 19:49:43.568731 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:43.568695 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.439878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:45.439829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.754747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:45.754660 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.950255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:46.949878 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.926691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:47.926654 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.387169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:48.387122 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.538123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:48.537503 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.917496 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:48.916448 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.185706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:49.185604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.395435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:50.395393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.136276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:52.136234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.347400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:52.347357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.507288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:52.507192 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.919070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:53.919032 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 19:49:53.923556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:53.923538 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:53.953051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:53.951350 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-f83282-openshift-default-dockercfg-4lzlr\"" Oct 30 19:49:53.957652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:53.957624 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 19:49:54.015653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.015617 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.015855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.015829 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.015969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.015895 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.015976 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmp8\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.016045 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.016105 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.016181 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.016223 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.016315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.016263 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.044748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.044717 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.117211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117178 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117237 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117409 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmp8\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117476 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117520 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117554 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117614 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117656 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117673 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117693 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117775 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.117990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.117864 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.118155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.118131 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.118284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.118253 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.119955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.119926 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.120294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.120273 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.134707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.134680 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.135031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.135010 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmp8\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.185715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.185645 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 19:49:54.191303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.191287 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.203720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.203679 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.208579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.208514 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 19:49:54.209828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.209808 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-5a0a17-dockercfg-hwt9z\"" Oct 30 19:49:54.209939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.209833 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-5a0a17\"" Oct 30 19:49:54.310979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.310939 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:54.372822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.322977 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.372822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.362957 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5r6v\" (UniqueName: \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.372822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.369613 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 19:49:54.382631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.382293 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 19:49:54.382631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.382462 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.464682 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.464644 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.464907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.464878 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.466386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.466347 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.466547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.466457 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5r6v\" (UniqueName: \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.466757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.466736 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq24\" (UniqueName: \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.486231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.486195 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5r6v\" (UniqueName: \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\") pod \"payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.492564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.487404 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:49:54.502713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.502688 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:49:54.506761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.506735 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:54.507375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.507352 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:49:54.531207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.531180 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:49:54.535300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.535192 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:49:54.572936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.568160 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq24\" (UniqueName: \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.572936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.568336 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.572936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.569247 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.591198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.590664 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq24\" (UniqueName: \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\") pod \"payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.720897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.720799 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:54.787647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.787344 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 19:49:54.796605 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:49:54.796564 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff90ce6_b0c6_4237_8714_36b0986e19ce.slice/crio-d8ccd8f629241d372e2a4d8b299b5a399951f653e8ab777a0a869d61174a614a WatchSource:0}: Error finding container d8ccd8f629241d372e2a4d8b299b5a399951f653e8ab777a0a869d61174a614a: Status 404 returned error can't find the container with id d8ccd8f629241d372e2a4d8b299b5a399951f653e8ab777a0a869d61174a614a Oct 30 19:49:54.800825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.800785 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:49:54.800972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.800886 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:49:54.800972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.800941 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:49:54.859660 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:49:54.859613 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49b6b2e_cb5e_4d0c_b443_2f7daa5189ee.slice/crio-5741e6c96f84d70235caf610155426b04d876f3aa1e37b94002e6352b2a74873 WatchSource:0}: Error finding container 5741e6c96f84d70235caf610155426b04d876f3aa1e37b94002e6352b2a74873: Status 404 returned error can't find the container with id 5741e6c96f84d70235caf610155426b04d876f3aa1e37b94002e6352b2a74873 Oct 30 19:49:54.877904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:54.877079 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 19:49:55.026019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.025944 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 19:49:55.033100 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:49:55.033039 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c8c57a_da7b_4b6d_91ec_a2f2d1ed8803.slice/crio-61edfa2ccc109659c05dcffa22dd0e0eaa3ddb1f214a13d46cb0e0892624b5ad WatchSource:0}: Error finding container 61edfa2ccc109659c05dcffa22dd0e0eaa3ddb1f214a13d46cb0e0892624b5ad: Status 404 returned error can't find the container with id 61edfa2ccc109659c05dcffa22dd0e0eaa3ddb1f214a13d46cb0e0892624b5ad Oct 30 19:49:55.626596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.626563 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" event={"ID":"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803","Type":"ContainerStarted","Data":"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c"} Oct 30 19:49:55.626809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.626623 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" event={"ID":"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803","Type":"ContainerStarted","Data":"61edfa2ccc109659c05dcffa22dd0e0eaa3ddb1f214a13d46cb0e0892624b5ad"} Oct 30 19:49:55.626809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.626656 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:49:55.628385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.628333 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" event={"ID":"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee","Type":"ContainerStarted","Data":"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088"} Oct 30 19:49:55.628526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.628388 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:49:55.628526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.628412 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" event={"ID":"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee","Type":"ContainerStarted","Data":"5741e6c96f84d70235caf610155426b04d876f3aa1e37b94002e6352b2a74873"} Oct 30 19:49:55.630156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.630127 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" event={"ID":"3ff90ce6-b0c6-4237-8714-36b0986e19ce","Type":"ContainerStarted","Data":"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4"} Oct 30 19:49:55.630268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.630184 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" event={"ID":"3ff90ce6-b0c6-4237-8714-36b0986e19ce","Type":"ContainerStarted","Data":"d8ccd8f629241d372e2a4d8b299b5a399951f653e8ab777a0a869d61174a614a"} Oct 30 19:49:55.663555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.663505 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" podStartSLOduration=1.6634930319999999 podStartE2EDuration="1.663493032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:55.659811373 +0000 UTC m=+2703.371336739" watchObservedRunningTime="2026-10-30 19:49:55.663493032 +0000 UTC m=+2703.375018398" Oct 30 19:49:55.700044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.699977 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" podStartSLOduration=1.699958376 podStartE2EDuration="1.699958376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:55.696875272 +0000 UTC m=+2703.408400639" watchObservedRunningTime="2026-10-30 19:49:55.699958376 +0000 UTC m=+2703.411483747" Oct 30 19:49:55.739328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.739272 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podStartSLOduration=2.739257278 podStartE2EDuration="2.739257278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:55.735343934 +0000 UTC m=+2703.446869300" watchObservedRunningTime="2026-10-30 19:49:55.739257278 +0000 UTC m=+2703.450782643" Oct 30 19:49:55.846471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:55.835163 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.311533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:56.311489 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:49:56.313546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:56.313515 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 19:49:56.313686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:56.313579 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 19:49:57.312451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:57.312414 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 19:49:57.313030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:57.312478 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 19:49:57.456415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:57.456376 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:57.608286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:57.608194 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.321114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:58.320867 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 19:49:58.321114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:58.320964 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 19:49:58.562692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:58.562656 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.048444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:59.048403 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.312308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:59.312145 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 19:49:59.312308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:59.312216 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 19:49:59.448203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:59.448166 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.701469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:49:59.701339 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:00.311656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:00.311620 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 19:50:00.311872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:00.311683 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 19:50:01.186695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:01.181069 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.513623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:01.513543 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:50:01.514593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:01.514563 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:50:01.515543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:01.515520 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 19:50:01.537031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:01.536988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:02.925050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:02.925011 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.519370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:03.519334 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.719825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:03.719785 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.938112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:04.937996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.101129 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:05.101075 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:50:05.429000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.428960 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.571406 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.571345 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:50:05.577696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.577654 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.581401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.581365 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.595124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.595069 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:50:05.597609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.597583 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-852709\"" Oct 30 19:50:05.597755 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.597662 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-852709-dockercfg-dplwd\"" Oct 30 19:50:05.754169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.750288 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.754169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.750580 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6js\" (UniqueName: \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.851992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.851936 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.852485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.852456 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6js\" (UniqueName: \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.854032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.853966 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.872869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.872800 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:50:05.883192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.879201 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:05.885524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.885303 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:50:05.888770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.888435 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6js\" (UniqueName: \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\") pod \"payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.901157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.900022 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:05.953718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.953678 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdc74\" (UniqueName: \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:05.954202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:05.953838 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.055458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.055111 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdc74\" (UniqueName: \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.055458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.055271 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.055967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.055944 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.072276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.072250 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdc74\" (UniqueName: \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\") pod \"payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.216427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.216400 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:50:06.218064 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:06.218037 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04d87cf_c70e_4048_b18e_faeaaa878a2e.slice/crio-2a78c0cfb78b0ceb3d137baa6a6919bb266ae96437104d50b17428b3755a3376 WatchSource:0}: Error finding container 2a78c0cfb78b0ceb3d137baa6a6919bb266ae96437104d50b17428b3755a3376: Status 404 returned error can't find the container with id 2a78c0cfb78b0ceb3d137baa6a6919bb266ae96437104d50b17428b3755a3376 Oct 30 19:50:06.250864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.250839 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:06.437129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.433138 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:06.617888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.592905 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:50:06.622247 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:06.622216 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd63e0b_5646_4faf_99cd_a0a93396ffd1.slice/crio-3e13fb017c7271f87e17794d52804e0088cde604dbda620414234ba79c6b0e6d WatchSource:0}: Error finding container 3e13fb017c7271f87e17794d52804e0088cde604dbda620414234ba79c6b0e6d: Status 404 returned error can't find the container with id 3e13fb017c7271f87e17794d52804e0088cde604dbda620414234ba79c6b0e6d Oct 30 19:50:06.646272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.645352 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 19:50:06.663220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.663188 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 19:50:06.872970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.872866 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" event={"ID":"4bd63e0b-5646-4faf-99cd-a0a93396ffd1","Type":"ContainerStarted","Data":"3e13fb017c7271f87e17794d52804e0088cde604dbda620414234ba79c6b0e6d"} Oct 30 19:50:06.877037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.876995 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" event={"ID":"e04d87cf-c70e-4048-b18e-faeaaa878a2e","Type":"ContainerStarted","Data":"57dfc5624f8e18fe054559c8a716545b7deead63af06fa5b9e0aa28f985543ce"} Oct 30 19:50:06.880724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.880694 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:06.880992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.880972 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" event={"ID":"e04d87cf-c70e-4048-b18e-faeaaa878a2e","Type":"ContainerStarted","Data":"2a78c0cfb78b0ceb3d137baa6a6919bb266ae96437104d50b17428b3755a3376"} Oct 30 19:50:06.923619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:06.922075 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" podStartSLOduration=1.9220543220000001 podStartE2EDuration="1.922054322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:06.917079135 +0000 UTC m=+2714.628604503" watchObservedRunningTime="2026-10-30 19:50:06.922054322 +0000 UTC m=+2714.633579689" Oct 30 19:50:07.127456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.127362 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.286835 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.286796 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.432520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.432428 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.739267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.739183 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.889733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.889693 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" event={"ID":"4bd63e0b-5646-4faf-99cd-a0a93396ffd1","Type":"ContainerStarted","Data":"b8d26eb87dad222e17026ea3b1eefa637266ebbe6429268ee3c4f8205a52b074"} Oct 30 19:50:07.889943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.889767 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:07.927539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:07.927486 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" podStartSLOduration=2.927471793 podStartE2EDuration="2.927471793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:07.924458678 +0000 UTC m=+2715.635984044" watchObservedRunningTime="2026-10-30 19:50:07.927471793 +0000 UTC m=+2715.638997158" Oct 30 19:50:09.141580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:09.141542 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.018010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:11.017976 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:12.874842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:12.874803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.996996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:13.994710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.145986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:14.145948 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.316640 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:14.316605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.589164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:14.586694 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.367779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.367724 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw"] Oct 30 19:50:16.387586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.387549 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.391955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.391919 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw"] Oct 30 19:50:16.401481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.401434 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:16.423920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.423889 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:50:16.436147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.432258 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.500257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.500210 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrxh\" (UniqueName: \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.500477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.500446 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.500595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.500574 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.601852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.601810 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrxh\" (UniqueName: \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.602063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.602016 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.602190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.602065 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.602867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.602842 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.604588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.604566 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.618811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.618743 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrxh\" (UniqueName: \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\") pod \"maas-api-revoke-sub-a132ac6a-v67xw\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:16.729940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:16.729903 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:17.042348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:17.042315 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw"] Oct 30 19:50:17.044458 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:17.044418 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9d1b39_e2d2_41bf_a5b9_625922d0115c.slice/crio-d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a WatchSource:0}: Error finding container d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a: Status 404 returned error can't find the container with id d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a Oct 30 19:50:17.791990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:17.791897 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.894258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:17.894224 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:50:18.069744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:18.069571 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" event={"ID":"1b9d1b39-e2d2-41bf-a5b9-625922d0115c","Type":"ContainerStarted","Data":"c406ff64b73aadaff294c862749dfa5878312a0eb012a17cfb257605835b25e3"} Oct 30 19:50:18.070203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:18.070155 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" event={"ID":"1b9d1b39-e2d2-41bf-a5b9-625922d0115c","Type":"ContainerStarted","Data":"d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a"} Oct 30 19:50:18.105845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:18.105784 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" podStartSLOduration=2.105767273 podStartE2EDuration="2.105767273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:18.102935913 +0000 UTC m=+2725.814461276" watchObservedRunningTime="2026-10-30 19:50:18.105767273 +0000 UTC m=+2725.817292638" Oct 30 19:50:18.189119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:18.185749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:18.902825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:18.902786 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:50:19.083748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:19.083711 4990 generic.go:356] "Generic (PLEG): container finished" podID="1b9d1b39-e2d2-41bf-a5b9-625922d0115c" containerID="c406ff64b73aadaff294c862749dfa5878312a0eb012a17cfb257605835b25e3" exitCode=0 Oct 30 19:50:19.083923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:19.083795 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" event={"ID":"1b9d1b39-e2d2-41bf-a5b9-625922d0115c","Type":"ContainerDied","Data":"c406ff64b73aadaff294c862749dfa5878312a0eb012a17cfb257605835b25e3"} Oct 30 19:50:20.282992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.282950 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:20.293143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.291748 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:20.398490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.398392 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\") pod \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " Oct 30 19:50:20.398681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.398662 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\") pod \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " Oct 30 19:50:20.398759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.398727 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\" (UniqueName: \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\") pod \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\" (UID: \"1b9d1b39-e2d2-41bf-a5b9-625922d0115c\") " Oct 30 19:50:20.401820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.399564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca" pod "1b9d1b39-e2d2-41bf-a5b9-625922d0115c" (UID: "1b9d1b39-e2d2-41bf-a5b9-625922d0115c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:20.402346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.402167 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh" pod "1b9d1b39-e2d2-41bf-a5b9-625922d0115c" (UID: "1b9d1b39-e2d2-41bf-a5b9-625922d0115c"). InnerVolumeSpecName "kube-api-access-qkrxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:20.402708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.402673 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca" pod "1b9d1b39-e2d2-41bf-a5b9-625922d0115c" (UID: "1b9d1b39-e2d2-41bf-a5b9-625922d0115c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:20.500746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.500706 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:20.500746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.500741 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:20.500746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:20.500750 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrxh\" (UniqueName: \"kubernetes.io/projected/1b9d1b39-e2d2-41bf-a5b9-625922d0115c-kube-api-access-qkrxh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:21.103222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:21.103184 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" event={"ID":"1b9d1b39-e2d2-41bf-a5b9-625922d0115c","Type":"ContainerDied","Data":"d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a"} Oct 30 19:50:21.103222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:21.103221 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw" Oct 30 19:50:21.103427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:21.103246 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d853c5ccc09086a11078b1f3fd34518e16cf157d7db8c1a0029a462f4e7b2a3a" Oct 30 19:50:22.032068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:22.031326 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.533459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:23.533418 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:25.377117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:25.372959 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:26.780995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.780960 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567"] Oct 30 19:50:26.781860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.781838 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1b9d1b39-e2d2-41bf-a5b9-625922d0115c" containerName="revoke-keys" Oct 30 19:50:26.781913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.781864 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9d1b39-e2d2-41bf-a5b9-625922d0115c" containerName="revoke-keys" Oct 30 19:50:26.782117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.782101 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9d1b39-e2d2-41bf-a5b9-625922d0115c" containerName="revoke-keys" Oct 30 19:50:26.785872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.785853 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:26.804331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.804303 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567"] Oct 30 19:50:26.809361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.809333 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:26.809489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.809428 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:50:26.916725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.916687 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9zp\" (UniqueName: \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:26.916947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.916924 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:26.917050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:26.917033 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.018448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.018409 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.018673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.018477 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.018673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.018666 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9zp\" (UniqueName: \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.019336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.019284 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.021303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.021273 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.034674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.034534 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.036020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.035992 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9zp\" (UniqueName: \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\") pod \"maas-api-revoke-sub-a4d2b811-jp567\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.099673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.099634 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:27.414828 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.414799 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567"] Oct 30 19:50:27.417146 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:27.417118 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0a5e5a_c2c3_4f67_bb11_594feca3a9cf.slice/crio-604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9 WatchSource:0}: Error finding container 604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9: Status 404 returned error can't find the container with id 604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9 Oct 30 19:50:27.690327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.690297 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:50:27.695667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.695645 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.719512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.719481 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:50:27.831853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.831816 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.831870 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwvb\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832131 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832164 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832197 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832256 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832361 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832450 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.832720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.832487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934007 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934056 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934104 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934142 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934190 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934264 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934301 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934375 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934410 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwvb\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934524 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934573 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934722 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.934804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.934773 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.935201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.935177 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.937164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.937143 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.937317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.937299 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.950908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.950886 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:27.951079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:27.951061 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwvb\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:28.027237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.027207 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:28.164489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.164443 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" event={"ID":"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf","Type":"ContainerStarted","Data":"4ae72e8ed880db291f109303767d50e54c2784f00fdfdf5cbc3201f1d0e3bf19"} Oct 30 19:50:28.164661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.164517 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" event={"ID":"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf","Type":"ContainerStarted","Data":"604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9"} Oct 30 19:50:28.203628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.203529 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" podStartSLOduration=2.203515373 podStartE2EDuration="2.203515373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.199727961 +0000 UTC m=+2735.911253327" watchObservedRunningTime="2026-10-30 19:50:28.203515373 +0000 UTC m=+2735.915040739" Oct 30 19:50:28.353768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.353737 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:50:28.354535 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:28.354507 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e27524_6b26_4afc_92ef_32af2448c923.slice/crio-901d5c79a509da70b2bb05f9010457fd908d73c8b3763c6e3977cd1733b39f34 WatchSource:0}: Error finding container 901d5c79a509da70b2bb05f9010457fd908d73c8b3763c6e3977cd1733b39f34: Status 404 returned error can't find the container with id 901d5c79a509da70b2bb05f9010457fd908d73c8b3763c6e3977cd1733b39f34 Oct 30 19:50:28.357710 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.357666 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:28.357803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.357758 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:28.357847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.357803 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:28.716785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:28.716740 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:29.202206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:29.202163 4990 generic.go:356] "Generic (PLEG): container finished" podID="0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" containerID="4ae72e8ed880db291f109303767d50e54c2784f00fdfdf5cbc3201f1d0e3bf19" exitCode=0 Oct 30 19:50:29.202702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:29.202252 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" event={"ID":"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf","Type":"ContainerDied","Data":"4ae72e8ed880db291f109303767d50e54c2784f00fdfdf5cbc3201f1d0e3bf19"} Oct 30 19:50:29.204472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:29.204445 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" event={"ID":"96e27524-6b26-4afc-92ef-32af2448c923","Type":"ContainerStarted","Data":"85a6fcda4a155de9b2828256f3144ffe464ac0daacbcfc5cfb569d8b2793cc64"} Oct 30 19:50:29.204596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:29.204497 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" event={"ID":"96e27524-6b26-4afc-92ef-32af2448c923","Type":"ContainerStarted","Data":"901d5c79a509da70b2bb05f9010457fd908d73c8b3763c6e3977cd1733b39f34"} Oct 30 19:50:29.297114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:29.293829 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podStartSLOduration=2.2938105970000002 podStartE2EDuration="2.293810597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:29.272784354 +0000 UTC m=+2736.984309720" watchObservedRunningTime="2026-10-30 19:50:29.293810597 +0000 UTC m=+2737.005335963" Oct 30 19:50:30.028193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.028157 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:30.029408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.029376 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.029586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.029421 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 19:50:30.442300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.442269 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:30.604771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.604683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.612138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.612066 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\" (UniqueName: \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\") pod \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " Oct 30 19:50:30.612330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.612312 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\") pod \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " Oct 30 19:50:30.612421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.612363 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\") pod \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\" (UID: \"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf\") " Oct 30 19:50:30.613629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.613595 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca" pod "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" (UID: "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:30.618048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.617764 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp" pod "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" (UID: "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf"). InnerVolumeSpecName "kube-api-access-2x9zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:30.625497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.625470 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca" pod "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" (UID: "0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:30.719120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.718620 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:30.719120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.718657 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:30.719120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.718675 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9zp\" (UniqueName: \"kubernetes.io/projected/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf-kube-api-access-2x9zp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:30.758321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:30.758237 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.027776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:31.027693 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.027776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:31.027751 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 19:50:31.225540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:31.225513 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" Oct 30 19:50:31.225736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:31.225556 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567" event={"ID":"0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf","Type":"ContainerDied","Data":"604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9"} Oct 30 19:50:31.225736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:31.225614 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604740a7cde200748dc299806d0892cde0b6883b079d73072f5a9c24bb18a7b9" Oct 30 19:50:32.028647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:32.028604 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.029164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:32.028676 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 19:50:32.551210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:32.551172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.027787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:33.027695 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.027787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:33.027773 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 19:50:34.027748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:34.027717 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.028143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:34.027771 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 19:50:34.250949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:34.250904 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:35.101135 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:35.101070 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:50:35.137666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:35.137629 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:35.138305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:35.138283 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:35.139634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:35.139615 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:50:36.056202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:36.056169 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:37.620137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:37.619569 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.275260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:39.272847 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.936053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:39.936001 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.363900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.363861 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.742462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.742375 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:50:40.743565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.743543 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" containerName="revoke-keys" Oct 30 19:50:40.743674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.743566 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" containerName="revoke-keys" Oct 30 19:50:40.743792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.743781 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" containerName="revoke-keys" Oct 30 19:50:40.749675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.749644 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.762455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.762425 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-aae4d8-openshift-default-dockercfg-5dmm9\"" Oct 30 19:50:40.779282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.779254 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:50:40.815103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815053 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815222 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815299 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815325 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815350 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815379 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815471 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkh7\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815596 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.815675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.815625 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.838590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.838563 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.916661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916625 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkh7\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.916874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916734 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.916874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916773 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.916874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916834 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916942 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.916997 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917019 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917043 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917071 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917161 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917211 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917303 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917411 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.917824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.917803 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.919805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.919775 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.919924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.919817 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.933877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.933853 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkh7\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:40.934087 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.934070 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\") pod \"e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:41.000169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:40.994637 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:41.093114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:41.092369 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:41.481922 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:41.481885 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dba8c98_73c7_4670_b719_7304469946ca.slice/crio-ebe6cc8fd8d11c4c1eeec38b5b43de62b78aa1f10c23e681153df0f9f020f6b0 WatchSource:0}: Error finding container ebe6cc8fd8d11c4c1eeec38b5b43de62b78aa1f10c23e681153df0f9f020f6b0: Status 404 returned error can't find the container with id ebe6cc8fd8d11c4c1eeec38b5b43de62b78aa1f10c23e681153df0f9f020f6b0 Oct 30 19:50:41.487160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:41.487074 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:41.487287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:41.487208 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:41.487287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:41.487251 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:50:41.490118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:41.490074 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:50:42.338852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.338815 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerStarted","Data":"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960"} Oct 30 19:50:42.339040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.338869 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerStarted","Data":"ebe6cc8fd8d11c4c1eeec38b5b43de62b78aa1f10c23e681153df0f9f020f6b0"} Oct 30 19:50:42.395169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.395087 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podStartSLOduration=2.395070655 podStartE2EDuration="2.395070655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:42.384742399 +0000 UTC m=+2750.096267766" watchObservedRunningTime="2026-10-30 19:50:42.395070655 +0000 UTC m=+2750.106596022" Oct 30 19:50:42.604478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.604370 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.741143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.740696 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.883426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:42.883328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:43.093663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:43.093621 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:43.099936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:43.099912 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:43.346788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:43.346758 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:43.347874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:43.347850 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:50:44.694398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:44.694363 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:44.853506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:44.853473 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.006872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:45.006786 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:50:45.007072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:45.007046 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" containerID="cri-o://6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0" gracePeriod=30 Oct 30 19:50:46.507930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:46.507877 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:46.672304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:46.672268 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:46.863114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:46.862979 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:47.113253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:47.112342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.175320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:48.175279 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.565503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:48.565460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.190902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.190859 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.344890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.344863 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:50:50.423729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.423696 4990 generic.go:356] "Generic (PLEG): container finished" podID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerID="6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0" exitCode=0 Oct 30 19:50:50.423919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.423763 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" Oct 30 19:50:50.423919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.423876 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerDied","Data":"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0"} Oct 30 19:50:50.424036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.423919 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb" event={"ID":"b4289f63-a454-4f75-9741-3bf4bf835fea","Type":"ContainerDied","Data":"b69233a04f7ad5f4f6685c86959376488e4674fbee5455e6ca143eb597eff0b1"} Oct 30 19:50:50.424036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.423948 4990 scope.go:122] "RemoveContainer" containerID="6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0" Oct 30 19:50:50.440474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.440448 4990 scope.go:122] "RemoveContainer" containerID="4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493" Oct 30 19:50:50.453404 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.453365 4990 scope.go:122] "RemoveContainer" containerID="6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0" Oct 30 19:50:50.453686 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:50.453665 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0\": container with ID starting with 6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0 not found: ID does not exist" containerID="6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0" Oct 30 19:50:50.453742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.453697 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0"} err="failed to get container status \"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0\": rpc error: code = NotFound desc = could not find container \"6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0\": container with ID starting with 6d8ac08cf99b423240d2d0cc914bc7dbe3b5d46411658e0697d07bb9bd8592a0 not found: ID does not exist" Oct 30 19:50:50.453742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.453714 4990 scope.go:122] "RemoveContainer" containerID="4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493" Oct 30 19:50:50.453970 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:50.453953 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493\": container with ID starting with 4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493 not found: ID does not exist" containerID="4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493" Oct 30 19:50:50.454015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.453975 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493"} err="failed to get container status \"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493\": rpc error: code = NotFound desc = could not find container \"4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493\": container with ID starting with 4a88254e7d0c1cfb3ec625609b0b357891cae4877196dceb1eef641442a8a493 not found: ID does not exist" Oct 30 19:50:50.517996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.517967 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518021 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518050 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518077 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518117 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518166 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518395 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518422 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:50.518502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518468 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:50.518633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518503 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518508 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:50.518633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518618 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:50.518761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518675 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\") pod \"b4289f63-a454-4f75-9741-3bf4bf835fea\" (UID: \"b4289f63-a454-4f75-9741-3bf4bf835fea\") " Oct 30 19:50:50.518761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.518725 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:50.519363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.519340 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.519363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.519360 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4289f63-a454-4f75-9741-3bf4bf835fea-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.519363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.519370 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.519689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.519378 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.519689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.519386 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.520472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.520442 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "kube-api-access-2ffbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:50.520664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.520645 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:50.520815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.520786 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:50.520949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.520932 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token" pod "b4289f63-a454-4f75-9741-3bf4bf835fea" (UID: "b4289f63-a454-4f75-9741-3bf4bf835fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:50.621870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.621782 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.621870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.621812 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.621870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.621827 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4289f63-a454-4f75-9741-3bf4bf835fea-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.621870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.621839 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffbg\" (UniqueName: \"kubernetes.io/projected/b4289f63-a454-4f75-9741-3bf4bf835fea-kube-api-access-2ffbg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:50.770885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.770843 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:50:50.782136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:50.782104 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-fgpcb"] Oct 30 19:50:51.977147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:51.972857 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.405247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:52.405212 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" path="/var/lib/kubelet/pods/b4289f63-a454-4f75-9741-3bf4bf835fea/volumes" Oct 30 19:50:53.530858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.515588 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:50:53.530858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.515811 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" containerID="cri-o://b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254" gracePeriod=30 Oct 30 19:50:53.643375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.615565 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:50:53.643375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.615802 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" containerID="cri-o://9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027" gracePeriod=30 Oct 30 19:50:53.643375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.641755 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:50:53.643375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.641983 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" containerID="cri-o://d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a" gracePeriod=30 Oct 30 19:50:53.760463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.759746 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:50:53.760463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:53.760030 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" containerID="cri-o://906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9" gracePeriod=30 Oct 30 19:50:54.299499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.299430 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:54.341415 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.341383 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515173 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515318 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515405 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515531 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515587 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515667 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.516895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.515698 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\") pod \"7e757b0d-22db-4c33-ad55-c3a6f9711735\" (UID: \"7e757b0d-22db-4c33-ad55-c3a6f9711735\") " Oct 30 19:50:54.527142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.523253 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:54.527142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.526547 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:54.531562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.529761 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:54.540177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.540066 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:54.541005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.540976 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "kube-api-access-w49kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:54.541148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.541082 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:54.541221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.541182 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert" pod "7e757b0d-22db-4c33-ad55-c3a6f9711735" (UID: "7e757b0d-22db-4c33-ad55-c3a6f9711735"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:54.543654 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543626 4990 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-kubeconfig\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543670 4990 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-cacerts\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543686 4990 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-ca-configmap\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543701 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543714 4990 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e757b0d-22db-4c33-ad55-c3a6f9711735-local-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543727 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49kr\" (UniqueName: \"kubernetes.io/projected/7e757b0d-22db-4c33-ad55-c3a6f9711735-kube-api-access-w49kr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.547246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.543757 4990 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e757b0d-22db-4c33-ad55-c3a6f9711735-istio-csr-dns-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:54.556134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.555187 4990 generic.go:356] "Generic (PLEG): container finished" podID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerID="b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254" exitCode=0 Oct 30 19:50:54.556134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.555306 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" event={"ID":"7e757b0d-22db-4c33-ad55-c3a6f9711735","Type":"ContainerDied","Data":"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254"} Oct 30 19:50:54.556134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.555353 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" event={"ID":"7e757b0d-22db-4c33-ad55-c3a6f9711735","Type":"ContainerDied","Data":"5fae26694fd735b82622cf1ccdcc23ad5fac855d95cb6192a7c68f3ab636a92a"} Oct 30 19:50:54.556134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.555382 4990 scope.go:122] "RemoveContainer" containerID="b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254" Oct 30 19:50:54.556134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.555530 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" Oct 30 19:50:54.572214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.572193 4990 scope.go:122] "RemoveContainer" containerID="b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254" Oct 30 19:50:54.572622 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:54.572599 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254\": container with ID starting with b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254 not found: ID does not exist" containerID="b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254" Oct 30 19:50:54.572709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.572633 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254"} err="failed to get container status \"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254\": rpc error: code = NotFound desc = could not find container \"b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254\": container with ID starting with b3a3aa64babb07c6ad42065ac9241a7c722410248e5e3f967a2e836d1786c254 not found: ID does not exist" Oct 30 19:50:54.609855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.609820 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:50:54.615192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:54.614669 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm"] Oct 30 19:50:56.277343 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:56.277301 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:56.403713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:56.403681 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" path="/var/lib/kubelet/pods/7e757b0d-22db-4c33-ad55-c3a6f9711735/volumes" Oct 30 19:50:56.819861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:56.819821 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-g2h62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:50:56.820025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:56.819891 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:50:57.567948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.567910 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85"] Oct 30 19:50:57.569053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569029 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:50:57.569125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569056 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:50:57.569125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569080 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" Oct 30 19:50:57.569125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569104 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" Oct 30 19:50:57.569381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569367 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" Oct 30 19:50:57.569416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.569393 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:50:57.574910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.574883 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.591962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.591937 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85"] Oct 30 19:50:57.596006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.595982 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:57.596190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.596007 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:50:57.698479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.698436 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.698692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.698567 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m74m\" (UniqueName: \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.698760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.698687 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.800136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.800081 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.800308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.800223 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m74m\" (UniqueName: \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.800308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.800292 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.801159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.801127 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.802822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.802797 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.827255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.827192 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m74m\" (UniqueName: \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\") pod \"maas-api-revoke-sub-4c5a237d-q2m85\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:57.895191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:57.895165 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:50:58.203486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.203404 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85"] Oct 30 19:50:58.207324 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:50:58.207281 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29950bf1_1294_47ba_908f_3c594a58b572.slice/crio-a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8 WatchSource:0}: Error finding container a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8: Status 404 returned error can't find the container with id a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8 Oct 30 19:50:58.233273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.233249 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-scx8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:50:58.233379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.233296 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:50:58.479771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.478018 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:58.590620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.590578 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" event={"ID":"29950bf1-1294-47ba-908f-3c594a58b572","Type":"ContainerStarted","Data":"db4c4ede185fb590d47298fe3cdb72b53b44a830a6d4d74715f0981f9be7d895"} Oct 30 19:50:58.591041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.590637 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" event={"ID":"29950bf1-1294-47ba-908f-3c594a58b572","Type":"ContainerStarted","Data":"a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8"} Oct 30 19:50:58.627887 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:58.627837 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" podStartSLOduration=1.627821672 podStartE2EDuration="1.627821672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:58.626008552 +0000 UTC m=+2766.337533920" watchObservedRunningTime="2026-10-30 19:50:58.627821672 +0000 UTC m=+2766.339347040" Oct 30 19:50:59.011078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.009044 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:50:59.017190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.016990 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:50:59.122522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122476 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.122720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122542 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.122720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122589 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.122720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122631 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.122893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122751 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.122893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122788 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.122893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122855 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.122893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122886 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122919 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.122981 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123039 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123071 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123123 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123187 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.123326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123226 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\") pod \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\" (UID: \"79a1adb9-d4a0-42eb-ac5e-4098b85bba68\") " Oct 30 19:50:59.123326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123283 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.123534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123339 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.123534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123402 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\") pod \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\" (UID: \"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6\") " Oct 30 19:50:59.123534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123427 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.123704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.123580 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.124311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.124047 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.124311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.124250 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.124520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.124441 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:59.124777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.124270 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.124777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.124608 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.125233 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.125609 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.125749 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.126228 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.126266 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.126611 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.127055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.126722 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.130577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.130542 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.130824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.130780 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:59.130824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.130789 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.131273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.131128 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.131273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.131171 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "kube-api-access-42jbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.131273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.131240 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg" pod "79a1adb9-d4a0-42eb-ac5e-4098b85bba68" (UID: "79a1adb9-d4a0-42eb-ac5e-4098b85bba68"). InnerVolumeSpecName "kube-api-access-j8ngg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.133859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.133830 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo" pod "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" (UID: "5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:59.139394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.139375 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:50:59.156633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.156607 4990 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nrwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:59.156728 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.156664 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nrwm" podUID="7e757b0d-22db-4c33-ad55-c3a6f9711735" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.113:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:59.225588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225553 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.225723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225632 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.225811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225795 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.225860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225850 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.225936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225906 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.225995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.225921 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.226055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226027 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.226145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226114 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.226207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226177 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.226296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.226359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226322 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\") pod \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\" (UID: \"2192b8c2-3fd8-4a75-9cee-bb925cef20f8\") " Oct 30 19:50:59.226359 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226334 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.226454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226371 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:59.226863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.226833 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.227027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227004 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227030 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227044 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227058 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227070 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227083 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227130 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227144 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227158 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227170 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ngg\" (UniqueName: \"kubernetes.io/projected/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-kube-api-access-j8ngg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227186 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227200 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jbr\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-kube-api-access-42jbr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227213 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227229 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227242 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227255 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227269 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227282 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227295 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a1adb9-d4a0-42eb-ac5e-4098b85bba68-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227309 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.227878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.227859 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.228465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.228449 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:59.228774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.228751 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:59.228827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.228777 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh" pod "2192b8c2-3fd8-4a75-9cee-bb925cef20f8" (UID: "2192b8c2-3fd8-4a75-9cee-bb925cef20f8"). InnerVolumeSpecName "kube-api-access-ghnlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:59.328778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.328741 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.328778 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.328774 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.329044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.328796 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.329044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.328806 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnlh\" (UniqueName: \"kubernetes.io/projected/2192b8c2-3fd8-4a75-9cee-bb925cef20f8-kube-api-access-ghnlh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:50:59.602189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.602101 4990 generic.go:356] "Generic (PLEG): container finished" podID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerID="906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9" exitCode=0 Oct 30 19:50:59.602189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.602177 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" Oct 30 19:50:59.602714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.602194 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" event={"ID":"2192b8c2-3fd8-4a75-9cee-bb925cef20f8","Type":"ContainerDied","Data":"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9"} Oct 30 19:50:59.602714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.602257 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8" event={"ID":"2192b8c2-3fd8-4a75-9cee-bb925cef20f8","Type":"ContainerDied","Data":"389ab394b0982f7040570d584209ca075747c6e4142dfbab6e9cd29aec085e6a"} Oct 30 19:50:59.602714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.602286 4990 scope.go:122] "RemoveContainer" containerID="906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9" Oct 30 19:50:59.604210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.604179 4990 generic.go:356] "Generic (PLEG): container finished" podID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerID="9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027" exitCode=0 Oct 30 19:50:59.604452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.604431 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" Oct 30 19:50:59.604644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.604624 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" event={"ID":"79a1adb9-d4a0-42eb-ac5e-4098b85bba68","Type":"ContainerDied","Data":"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027"} Oct 30 19:50:59.604725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.604674 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62" event={"ID":"79a1adb9-d4a0-42eb-ac5e-4098b85bba68","Type":"ContainerDied","Data":"76dc5fa3734cf12069d672305862902d36d99ae2147dab27520c4493515e71a7"} Oct 30 19:50:59.606743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.606721 4990 generic.go:356] "Generic (PLEG): container finished" podID="29950bf1-1294-47ba-908f-3c594a58b572" containerID="db4c4ede185fb590d47298fe3cdb72b53b44a830a6d4d74715f0981f9be7d895" exitCode=0 Oct 30 19:50:59.606850 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.606830 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" event={"ID":"29950bf1-1294-47ba-908f-3c594a58b572","Type":"ContainerDied","Data":"db4c4ede185fb590d47298fe3cdb72b53b44a830a6d4d74715f0981f9be7d895"} Oct 30 19:50:59.608711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.608687 4990 generic.go:356] "Generic (PLEG): container finished" podID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerID="d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a" exitCode=0 Oct 30 19:50:59.608819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.608766 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" event={"ID":"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6","Type":"ContainerDied","Data":"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a"} Oct 30 19:50:59.608819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.608807 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" event={"ID":"5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6","Type":"ContainerDied","Data":"387b00d21821f7cf312fd720e5bcbbb16ba396f566282894640359d0014b950f"} Oct 30 19:50:59.608939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.608890 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v" Oct 30 19:50:59.624911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.624891 4990 scope.go:122] "RemoveContainer" containerID="906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9" Oct 30 19:50:59.625303 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:59.625281 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9\": container with ID starting with 906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9 not found: ID does not exist" containerID="906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9" Oct 30 19:50:59.625434 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.625411 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9"} err="failed to get container status \"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9\": rpc error: code = NotFound desc = could not find container \"906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9\": container with ID starting with 906182d9819237fa2b6962ad2f0b90aa51317faa2f70b847e90e2f4217071dd9 not found: ID does not exist" Oct 30 19:50:59.625711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.625690 4990 scope.go:122] "RemoveContainer" containerID="9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027" Oct 30 19:50:59.639458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.639436 4990 scope.go:122] "RemoveContainer" containerID="9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027" Oct 30 19:50:59.639856 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:59.639808 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027\": container with ID starting with 9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027 not found: ID does not exist" containerID="9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027" Oct 30 19:50:59.639954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.639865 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027"} err="failed to get container status \"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027\": rpc error: code = NotFound desc = could not find container \"9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027\": container with ID starting with 9e0cdd6d361ce940e2fa293e051e0fe5d9513921d1ce61084e634bfe58ca9027 not found: ID does not exist" Oct 30 19:50:59.639954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.639898 4990 scope.go:122] "RemoveContainer" containerID="d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a" Oct 30 19:50:59.653179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.653158 4990 scope.go:122] "RemoveContainer" containerID="d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a" Oct 30 19:50:59.653474 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:50:59.653454 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a\": container with ID starting with d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a not found: ID does not exist" containerID="d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a" Oct 30 19:50:59.653555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.653481 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a"} err="failed to get container status \"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a\": rpc error: code = NotFound desc = could not find container \"d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a\": container with ID starting with d6b29f3272242039d080651764a5828700eef382ad49c6668438744cce02276a not found: ID does not exist" Oct 30 19:50:59.711766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.711733 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:50:59.717162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.717136 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nlrv8"] Oct 30 19:50:59.741874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.741843 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:50:59.747935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.747905 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-g2h62"] Oct 30 19:50:59.779289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.779257 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:50:59.783476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:50:59.783446 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-scx8v"] Oct 30 19:51:00.358381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.358349 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:00.405152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.405119 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" path="/var/lib/kubelet/pods/2192b8c2-3fd8-4a75-9cee-bb925cef20f8/volumes" Oct 30 19:51:00.405711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.405695 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" path="/var/lib/kubelet/pods/5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6/volumes" Oct 30 19:51:00.406154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.406139 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" path="/var/lib/kubelet/pods/79a1adb9-d4a0-42eb-ac5e-4098b85bba68/volumes" Oct 30 19:51:00.810860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.810833 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:51:00.951960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.951873 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\") pod \"29950bf1-1294-47ba-908f-3c594a58b572\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " Oct 30 19:51:00.952185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.951988 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\" (UniqueName: \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\") pod \"29950bf1-1294-47ba-908f-3c594a58b572\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " Oct 30 19:51:00.952185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.952082 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\") pod \"29950bf1-1294-47ba-908f-3c594a58b572\" (UID: \"29950bf1-1294-47ba-908f-3c594a58b572\") " Oct 30 19:51:00.952363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.952335 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca" pod "29950bf1-1294-47ba-908f-3c594a58b572" (UID: "29950bf1-1294-47ba-908f-3c594a58b572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:00.952760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.952736 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:00.954132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.954081 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca" pod "29950bf1-1294-47ba-908f-3c594a58b572" (UID: "29950bf1-1294-47ba-908f-3c594a58b572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:00.954230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:00.954211 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m" pod "29950bf1-1294-47ba-908f-3c594a58b572" (UID: "29950bf1-1294-47ba-908f-3c594a58b572"). InnerVolumeSpecName "kube-api-access-2m74m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:01.054658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.054627 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29950bf1-1294-47ba-908f-3c594a58b572-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:01.054658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.054653 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m74m\" (UniqueName: \"kubernetes.io/projected/29950bf1-1294-47ba-908f-3c594a58b572-kube-api-access-2m74m\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:01.634081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.634051 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" Oct 30 19:51:01.634298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.634083 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85" event={"ID":"29950bf1-1294-47ba-908f-3c594a58b572","Type":"ContainerDied","Data":"a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8"} Oct 30 19:51:01.634298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.634160 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ddc42a689c220a85f436ac7681e54d3985209ab78ee1c7945e981a86e2f1f8" Oct 30 19:51:01.667473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:01.667440 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:02.525052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:02.525013 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:04.196285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:04.195594 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.097777 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:51:05.097736 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:51:05.757306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:05.757266 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.540351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:07.540312 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:08.190523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.190492 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb"] Oct 30 19:51:08.191192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191177 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" Oct 30 19:51:08.191264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191193 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" Oct 30 19:51:08.191264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191227 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:51:08.191264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191234 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:51:08.191264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191254 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" Oct 30 19:51:08.191264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191262 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191272 4990 cpu_manager.go:414] "containerMap: removing container" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191277 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191285 4990 cpu_manager.go:414] "containerMap: removing container" podUID="29950bf1-1294-47ba-908f-3c594a58b572" containerName="revoke-keys" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191290 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29950bf1-1294-47ba-908f-3c594a58b572" containerName="revoke-keys" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191397 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2192b8c2-3fd8-4a75-9cee-bb925cef20f8" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191407 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a1adb9-d4a0-42eb-ac5e-4098b85bba68" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191415 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4289f63-a454-4f75-9741-3bf4bf835fea" containerName="istio-proxy" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191423 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="29950bf1-1294-47ba-908f-3c594a58b572" containerName="revoke-keys" Oct 30 19:51:08.191429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.191428 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd6d3e4-dc25-49d8-8d14-155f82e0b3b6" containerName="istio-proxy" Oct 30 19:51:08.195225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.195209 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.212870 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.212834 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb"] Oct 30 19:51:08.217423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.217400 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:08.217565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.217473 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:51:08.260183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.260148 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.260413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.260269 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prktx\" (UniqueName: \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.260494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.260473 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.361924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.361885 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.362130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.361961 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prktx\" (UniqueName: \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.362222 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.362202 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.362800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.362779 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.364302 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.364287 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.379357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.379330 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prktx\" (UniqueName: \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\") pod \"maas-api-revoke-sub-1f0b648a-6cptb\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.510212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.510134 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:08.821241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:08.821216 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb"] Oct 30 19:51:08.822900 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:08.822875 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df6810e_f549_4d9e_8cb0_b1abeb8847cc.slice/crio-39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c WatchSource:0}: Error finding container 39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c: Status 404 returned error can't find the container with id 39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c Oct 30 19:51:09.681350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:09.681315 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:09.704982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:09.704927 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" event={"ID":"3df6810e-f549-4d9e-8cb0-b1abeb8847cc","Type":"ContainerStarted","Data":"89bedba5fb9f3e7c3d7aa3a95f9e0a128bf6e41e3a808f4cdb47e8da44120183"} Oct 30 19:51:09.705205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:09.705012 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" event={"ID":"3df6810e-f549-4d9e-8cb0-b1abeb8847cc","Type":"ContainerStarted","Data":"39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c"} Oct 30 19:51:09.758284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:09.758224 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" podStartSLOduration=1.758206256 podStartE2EDuration="1.758206256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:09.751331067 +0000 UTC m=+2777.462856432" watchObservedRunningTime="2026-10-30 19:51:09.758206256 +0000 UTC m=+2777.469731627" Oct 30 19:51:10.716941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:10.716904 4990 generic.go:356] "Generic (PLEG): container finished" podID="3df6810e-f549-4d9e-8cb0-b1abeb8847cc" containerID="89bedba5fb9f3e7c3d7aa3a95f9e0a128bf6e41e3a808f4cdb47e8da44120183" exitCode=0 Oct 30 19:51:10.717378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:10.716987 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" event={"ID":"3df6810e-f549-4d9e-8cb0-b1abeb8847cc","Type":"ContainerDied","Data":"89bedba5fb9f3e7c3d7aa3a95f9e0a128bf6e41e3a808f4cdb47e8da44120183"} Oct 30 19:51:11.646883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:11.646845 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:11.902305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:11.902242 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:12.007864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.007820 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\") pod \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " Oct 30 19:51:12.008076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.007959 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\" (UniqueName: \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\") pod \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " Oct 30 19:51:12.008076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.007999 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\") pod \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\" (UID: \"3df6810e-f549-4d9e-8cb0-b1abeb8847cc\") " Oct 30 19:51:12.008456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.008424 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca" pod "3df6810e-f549-4d9e-8cb0-b1abeb8847cc" (UID: "3df6810e-f549-4d9e-8cb0-b1abeb8847cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:12.008672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.008632 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:12.010144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.010073 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca" pod "3df6810e-f549-4d9e-8cb0-b1abeb8847cc" (UID: "3df6810e-f549-4d9e-8cb0-b1abeb8847cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:12.010250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.010176 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx" pod "3df6810e-f549-4d9e-8cb0-b1abeb8847cc" (UID: "3df6810e-f549-4d9e-8cb0-b1abeb8847cc"). InnerVolumeSpecName "kube-api-access-prktx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:12.109669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.109636 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prktx\" (UniqueName: \"kubernetes.io/projected/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-kube-api-access-prktx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:12.109669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.109663 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6810e-f549-4d9e-8cb0-b1abeb8847cc-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:12.741897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.741817 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" Oct 30 19:51:12.742079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.741813 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb" event={"ID":"3df6810e-f549-4d9e-8cb0-b1abeb8847cc","Type":"ContainerDied","Data":"39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c"} Oct 30 19:51:12.742079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:12.741944 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39789cfa8167c4d53d6580b855c2e292b1092e55c71c5142268011bb736b0e6c" Oct 30 19:51:13.294323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.294263 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.885072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.885039 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 19:51:13.885791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.885767 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3df6810e-f549-4d9e-8cb0-b1abeb8847cc" containerName="revoke-keys" Oct 30 19:51:13.885791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.885786 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df6810e-f549-4d9e-8cb0-b1abeb8847cc" containerName="revoke-keys" Oct 30 19:51:13.886008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.885992 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df6810e-f549-4d9e-8cb0-b1abeb8847cc" containerName="revoke-keys" Oct 30 19:51:13.890016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.889995 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:13.903249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.903231 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a56cb5-openshift-default-dockercfg-2jc2r\"" Oct 30 19:51:13.918950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:13.918920 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 19:51:14.035733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035697 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.035924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035753 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.035924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035783 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.035924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035812 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.036046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035912 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.036046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.035985 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.036153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.036068 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.036153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.036134 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4z5x\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.036240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.036220 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138103 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138141 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138163 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138348 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138394 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138507 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138631 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.138791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138766 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138971 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.138976 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.139026 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4z5x\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.139130 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.139345 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.139547 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.139521 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.140560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.140540 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.140972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.140954 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.155792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.155763 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4z5x\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.155792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.155766 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.205908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.205888 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:14.528844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.528812 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 19:51:14.530285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:14.530257 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f698579_c391_403d_923a_ff69ac928927.slice/crio-7c9cab264b5c88486cf72baf2e707b3baa66f4a07202db246847de93c72e7317 WatchSource:0}: Error finding container 7c9cab264b5c88486cf72baf2e707b3baa66f4a07202db246847de93c72e7317: Status 404 returned error can't find the container with id 7c9cab264b5c88486cf72baf2e707b3baa66f4a07202db246847de93c72e7317 Oct 30 19:51:14.532941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.532908 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:14.533017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.532976 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:14.533017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.533004 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:14.776121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.774920 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" event={"ID":"4f698579-c391-403d-923a-ff69ac928927","Type":"ContainerStarted","Data":"5e219ec191cdbafce96d7915da1fcb353f17ed81377827d6a978d5bfec68e37a"} Oct 30 19:51:14.776121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.774986 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" event={"ID":"4f698579-c391-403d-923a-ff69ac928927","Type":"ContainerStarted","Data":"7c9cab264b5c88486cf72baf2e707b3baa66f4a07202db246847de93c72e7317"} Oct 30 19:51:14.820320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:14.817936 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podStartSLOduration=1.817919464 podStartE2EDuration="1.817919464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.8148085 +0000 UTC m=+2782.526333866" watchObservedRunningTime="2026-10-30 19:51:14.817919464 +0000 UTC m=+2782.529444829" Oct 30 19:51:15.208722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.207678 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:15.209488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.209293 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.209488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.209348 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:15.210154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.210128 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.937822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.937786 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 19:51:15.943188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.943165 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:15.955708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:15.955685 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-095f55-openshift-default-dockercfg-ljw6s\"" Oct 30 19:51:16.007540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.007369 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088076 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088162 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088194 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088233 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088277 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088326 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088441 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088523 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8q2\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.089114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.088571 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.192966 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193066 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8q2\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193131 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193281 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193313 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193342 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193382 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193426 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193478 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.193841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.193702 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.196112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.194147 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.196112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.194426 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.196112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.194673 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.196112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.195401 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.197798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.197772 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.198026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.198010 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.207050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.207026 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.207181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.207079 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:16.211697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.211673 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.212057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.212035 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8q2\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.257570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.257542 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:16.444814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.444723 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:51:16.453114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.451767 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.500998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.500955 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:51:16.502645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.501543 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"kube-root-ca.crt\"" Oct 30 19:51:16.502645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.502136 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"sub-test-model-095f55-kserve-self-signed-certs\"" Oct 30 19:51:16.502645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.502248 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"default-dockercfg-jpj75\"" Oct 30 19:51:16.502645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.502321 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"openshift-service-ca.crt\"" Oct 30 19:51:16.543036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.542994 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:16.607069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607030 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnzv\" (UniqueName: \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.607235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607104 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.607235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607171 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.607357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607286 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.607487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607462 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.607600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.607582 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.608124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.608085 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 19:51:16.608574 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:16.608549 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f58063_a9e0_43bb_ad80_7a0b866c6bcd.slice/crio-b826e1a388689d39e732b82b2a0e77edb6d8ac0edb6bb5893d4917e51d63642e WatchSource:0}: Error finding container b826e1a388689d39e732b82b2a0e77edb6d8ac0edb6bb5893d4917e51d63642e: Status 404 returned error can't find the container with id b826e1a388689d39e732b82b2a0e77edb6d8ac0edb6bb5893d4917e51d63642e Oct 30 19:51:16.611599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.611566 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:16.611691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.611635 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:16.611691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.611666 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:16.769223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769179 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnzv\" (UniqueName: \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.769470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769245 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.769470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769273 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.769470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769385 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.769651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769515 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.769651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.769614 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.802117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.797555 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.802117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.798272 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.802117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.798782 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.831219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.829443 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.831219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.831025 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnzv\" (UniqueName: \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.831469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.831376 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\") pod \"sub-test-model-095f55-kserve-cd9999d85-p7pkd\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:16.834882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.834852 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" event={"ID":"31f58063-a9e0-43bb-ad80-7a0b866c6bcd","Type":"ContainerStarted","Data":"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10"} Oct 30 19:51:16.835144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.835121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" event={"ID":"31f58063-a9e0-43bb-ad80-7a0b866c6bcd","Type":"ContainerStarted","Data":"b826e1a388689d39e732b82b2a0e77edb6d8ac0edb6bb5893d4917e51d63642e"} Oct 30 19:51:16.907800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:16.907707 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podStartSLOduration=1.9076817689999999 podStartE2EDuration="1.907681769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:16.890858584 +0000 UTC m=+2784.602383951" watchObservedRunningTime="2026-10-30 19:51:16.907681769 +0000 UTC m=+2784.619207137" Oct 30 19:51:17.108855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.108815 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:17.241748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.241697 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.264277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.264036 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:17.264537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.261031 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:17.289164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.263573 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.289164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.264849 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:17.305348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.293184 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:17.526867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.513899 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:51:17.532269 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:17.532219 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod566631b8_0dd8_4d0a_bae4_5c93e161fed7.slice/crio-c3ec9b3f958757a4f1aa225cb1071e9731ab880fe45f2bb5f5b7cab8620f8340 WatchSource:0}: Error finding container c3ec9b3f958757a4f1aa225cb1071e9731ab880fe45f2bb5f5b7cab8620f8340: Status 404 returned error can't find the container with id c3ec9b3f958757a4f1aa225cb1071e9731ab880fe45f2bb5f5b7cab8620f8340 Oct 30 19:51:17.853991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.853953 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" event={"ID":"566631b8-0dd8-4d0a-bae4-5c93e161fed7","Type":"ContainerStarted","Data":"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7"} Oct 30 19:51:17.854327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.854001 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" event={"ID":"566631b8-0dd8-4d0a-bae4-5c93e161fed7","Type":"ContainerStarted","Data":"c3ec9b3f958757a4f1aa225cb1071e9731ab880fe45f2bb5f5b7cab8620f8340"} Oct 30 19:51:17.920155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:17.917748 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" podStartSLOduration=1.917729035 podStartE2EDuration="1.917729035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:17.917630833 +0000 UTC m=+2785.629156202" watchObservedRunningTime="2026-10-30 19:51:17.917729035 +0000 UTC m=+2785.629254402" Oct 30 19:51:18.109781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.109648 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:18.213117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.213047 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.213417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.213385 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:18.258008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.257975 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.258236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.258164 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:18.840719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.840676 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:51:18.864992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.864963 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:18.930159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.928136 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-287ea2\"/\"kube-root-ca.crt\"" Oct 30 19:51:18.930159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.928468 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-287ea2\"/\"auth-test-model-287ea2-kserve-self-signed-certs\"" Oct 30 19:51:18.936129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.934813 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:18.945232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.943271 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-287ea2\"/\"default-dockercfg-vkqfl\"" Oct 30 19:51:18.956522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:18.956486 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:51:19.013120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.012462 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-287ea2\"/\"openshift-service-ca.crt\"" Oct 30 19:51:19.060380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060341 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.060598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060512 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.060598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060557 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.060598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060573 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.060777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060636 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.060777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.060677 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5v4l\" (UniqueName: \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.093175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.084698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188583 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188680 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188707 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188795 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188850 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5v4l\" (UniqueName: \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.188938 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.189997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.189959 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.204952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.204916 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.205589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.205394 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.210003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.209522 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.210003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.209615 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:19.256119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.244550 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.256119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.254004 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5v4l\" (UniqueName: \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.256119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.254592 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\") pod \"auth-test-model-287ea2-kserve-899f4d757-wcwb9\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.257981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.257951 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.258142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.258002 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:19.283115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.272075 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:19.318118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.317386 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:19.736270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.735044 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:51:19.957521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.957487 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" event={"ID":"78e67769-a212-42da-8082-1f1c56129c31","Type":"ContainerStarted","Data":"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148"} Oct 30 19:51:19.957760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:19.957744 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" event={"ID":"78e67769-a212-42da-8082-1f1c56129c31","Type":"ContainerStarted","Data":"f2838faf37b4e26c99351615e203908c59a8b4c1286acd09bac79fc0c355e61d"} Oct 30 19:51:20.210115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.206522 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.210115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.206583 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 19:51:20.258486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.258449 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.258672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.258506 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:20.321221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.321180 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:20.549188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.549080 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.708041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:20.705105 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:21.259035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.258981 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:21.259573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.259070 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:21.420923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.420887 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:21.421868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.421689 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:21.423313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.423292 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 19:51:21.472117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:21.469626 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" podStartSLOduration=3.469604293 podStartE2EDuration="3.469604293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:20.005337469 +0000 UTC m=+2787.716862836" watchObservedRunningTime="2026-10-30 19:51:21.469604293 +0000 UTC m=+2789.181129659" Oct 30 19:51:22.258252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:22.258217 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:22.258445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:22.258272 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:22.383116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:22.382607 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:23.259024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:23.258984 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Oct 30 19:51:23.259273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:23.259051 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Oct 30 19:51:23.891689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:23.891499 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:24.264322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:24.264229 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:24.264759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:24.264733 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:24.265578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:24.265556 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 19:51:25.452038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:25.452000 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:27.109650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.109608 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:27.125212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.125177 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:27.152716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.152677 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:51:27.157813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.157788 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.170789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.170746 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-aae4d8-serving-cert\"" Oct 30 19:51:27.179245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.179212 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:51:27.247118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.247066 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.247354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.247251 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.247420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.247376 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddfpg\" (UniqueName: \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.247524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.247504 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.308427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.308386 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:27.348570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.348530 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.348761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.348655 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.348761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.348717 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.348875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.348787 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddfpg\" (UniqueName: \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.349385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.349358 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.351317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.351288 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.351407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.351326 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.369449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.369380 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddfpg\" (UniqueName: \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\") pod \"maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.476161 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.476075 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:27.476597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.476565 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:27.810001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:27.809968 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:51:27.812181 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:27.812073 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551cf2ae_f52e_4a69_8f13_039ecfa0ab65.slice/crio-0432f412b08d0c6daf62f7e1c44b1338684275146ef75096018fbc80c5df40e4 WatchSource:0}: Error finding container 0432f412b08d0c6daf62f7e1c44b1338684275146ef75096018fbc80c5df40e4: Status 404 returned error can't find the container with id 0432f412b08d0c6daf62f7e1c44b1338684275146ef75096018fbc80c5df40e4 Oct 30 19:51:28.055561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:28.055526 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" event={"ID":"551cf2ae-f52e-4a69-8f13-039ecfa0ab65","Type":"ContainerStarted","Data":"0432f412b08d0c6daf62f7e1c44b1338684275146ef75096018fbc80c5df40e4"} Oct 30 19:51:28.068751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:28.068655 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:51:29.067044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.067002 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" event={"ID":"551cf2ae-f52e-4a69-8f13-039ecfa0ab65","Type":"ContainerStarted","Data":"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe"} Oct 30 19:51:29.067604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.067276 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:29.103509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.103452 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" podStartSLOduration=1.76973803 podStartE2EDuration="2.103438082s" totalImagesPullingTime="333.700052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:29.10327366 +0000 UTC m=+2796.814799028" watchObservedRunningTime="2026-10-30 19:51:29.103438082 +0000 UTC m=+2796.814963447" Oct 30 19:51:29.207286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.207244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:29.318816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.318732 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:29.331412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:29.331377 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:30.093201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:30.093162 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:51:30.818699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:30.818142 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.391219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:32.391127 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.104129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:34.104061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.010980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.010933 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:51:35.042793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.042757 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:51:35.085797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.085762 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:51:35.115511 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:51:35.115469 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:51:35.216725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.216669 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.534804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.534768 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.760024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:35.759981 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.337675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:37.337604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.735160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:37.735062 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:51:37.819311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:37.819266 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:38.508769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.508725 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:38.793235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.793141 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 19:51:38.799981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.799881 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.833119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.832488 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ed2ede-openshift-default-dockercfg-bmbqc\"" Oct 30 19:51:38.833119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.832695 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 19:51:38.888074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888030 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888126 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888252 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888347 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2xl\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888572 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888652 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888676 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888807 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.888905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.888851 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:38.940163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.924803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:38.999611 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.001148 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.000153 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.001835 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.001983 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.002107 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2xl\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003080 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.002211 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003221 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003245 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003292 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003342 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003686 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.008113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.003898 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.076182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.045221 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.076182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.066905 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.089117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.081608 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2xl\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.089117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.082302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.100125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.091640 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\") pod \"e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.164677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.164640 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:39.437081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.436943 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.512716 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.512679 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 19:51:39.514490 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:39.514461 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd691bb5f_4b13_4d10_9349_c4666c2c0f26.slice/crio-e620e6c3ca2cae0d8a79464cb83b139017f6420af6e60404d12eee2d85cc35fe WatchSource:0}: Error finding container e620e6c3ca2cae0d8a79464cb83b139017f6420af6e60404d12eee2d85cc35fe: Status 404 returned error can't find the container with id e620e6c3ca2cae0d8a79464cb83b139017f6420af6e60404d12eee2d85cc35fe Oct 30 19:51:39.517676 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.517635 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:39.517797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.517727 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:39.517797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.517773 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:51:39.718985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:39.718929 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.015172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.014941 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" podUID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" containerName="payload-processing" containerID="cri-o://b8d26eb87dad222e17026ea3b1eefa637266ebbe6429268ee3c4f8205a52b074" gracePeriod=25 Oct 30 19:51:40.049235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.049191 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" podUID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" containerName="payload-pre-processing" containerID="cri-o://57dfc5624f8e18fe054559c8a716545b7deead63af06fa5b9e0aa28f985543ce" gracePeriod=25 Oct 30 19:51:40.318049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.318012 4990 generic.go:356] "Generic (PLEG): container finished" podID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" containerID="b8d26eb87dad222e17026ea3b1eefa637266ebbe6429268ee3c4f8205a52b074" exitCode=0 Oct 30 19:51:40.344114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.322431 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" event={"ID":"4bd63e0b-5646-4faf-99cd-a0a93396ffd1","Type":"ContainerDied","Data":"b8d26eb87dad222e17026ea3b1eefa637266ebbe6429268ee3c4f8205a52b074"} Oct 30 19:51:40.376491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.376456 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.377423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.377396 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerStarted","Data":"12755a23009f83152b1610371f670824badf48a24b7ea796e9eb442c67a73ea8"} Oct 30 19:51:40.377670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.377633 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerStarted","Data":"e620e6c3ca2cae0d8a79464cb83b139017f6420af6e60404d12eee2d85cc35fe"} Oct 30 19:51:40.790116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:40.775902 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podStartSLOduration=2.7758814000000003 podStartE2EDuration="2.7758814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:40.624576902 +0000 UTC m=+2808.336102267" watchObservedRunningTime="2026-10-30 19:51:40.7758814 +0000 UTC m=+2808.487406767" Oct 30 19:51:41.293455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.253997 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:41.364118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.352059 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:41.504539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.504450 4990 generic.go:356] "Generic (PLEG): container finished" podID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" containerID="57dfc5624f8e18fe054559c8a716545b7deead63af06fa5b9e0aa28f985543ce" exitCode=0 Oct 30 19:51:41.506212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.506181 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" event={"ID":"e04d87cf-c70e-4048-b18e-faeaaa878a2e","Type":"ContainerDied","Data":"57dfc5624f8e18fe054559c8a716545b7deead63af06fa5b9e0aa28f985543ce"} Oct 30 19:51:41.506422 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.506407 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:41.507878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.507860 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:51:41.541393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.535609 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:51:41.648681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.647749 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:51:41.698793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.698752 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\" (UniqueName: \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\") pod \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " Oct 30 19:51:41.699000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.698964 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\") pod \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\" (UID: \"4bd63e0b-5646-4faf-99cd-a0a93396ffd1\") " Oct 30 19:51:41.699430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.699392 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume" pod "4bd63e0b-5646-4faf-99cd-a0a93396ffd1" (UID: "4bd63e0b-5646-4faf-99cd-a0a93396ffd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:41.699765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.699745 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:41.701610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.701577 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74" pod "4bd63e0b-5646-4faf-99cd-a0a93396ffd1" (UID: "4bd63e0b-5646-4faf-99cd-a0a93396ffd1"). InnerVolumeSpecName "kube-api-access-jdc74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:41.800722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.800693 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\") pod \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " Oct 30 19:51:41.801313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.800752 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\" (UniqueName: \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\") pod \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\" (UID: \"e04d87cf-c70e-4048-b18e-faeaaa878a2e\") " Oct 30 19:51:41.801313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.801203 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume" pod "e04d87cf-c70e-4048-b18e-faeaaa878a2e" (UID: "e04d87cf-c70e-4048-b18e-faeaaa878a2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:41.801741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.801714 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdc74\" (UniqueName: \"kubernetes.io/projected/4bd63e0b-5646-4faf-99cd-a0a93396ffd1-kube-api-access-jdc74\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:41.801795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.801748 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04d87cf-c70e-4048-b18e-faeaaa878a2e-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:41.803156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.803135 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js" pod "e04d87cf-c70e-4048-b18e-faeaaa878a2e" (UID: "e04d87cf-c70e-4048-b18e-faeaaa878a2e"). InnerVolumeSpecName "kube-api-access-ck6js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:41.833121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.823787 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:41.902984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:41.902948 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6js\" (UniqueName: \"kubernetes.io/projected/e04d87cf-c70e-4048-b18e-faeaaa878a2e-kube-api-access-ck6js\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:42.514075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.513990 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" Oct 30 19:51:42.514075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.514029 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2" event={"ID":"4bd63e0b-5646-4faf-99cd-a0a93396ffd1","Type":"ContainerDied","Data":"3e13fb017c7271f87e17794d52804e0088cde604dbda620414234ba79c6b0e6d"} Oct 30 19:51:42.514377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.514127 4990 scope.go:122] "RemoveContainer" containerID="b8d26eb87dad222e17026ea3b1eefa637266ebbe6429268ee3c4f8205a52b074" Oct 30 19:51:42.515976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.515944 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" Oct 30 19:51:42.516125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.516002 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft" event={"ID":"e04d87cf-c70e-4048-b18e-faeaaa878a2e","Type":"ContainerDied","Data":"2a78c0cfb78b0ceb3d137baa6a6919bb266ae96437104d50b17428b3755a3376"} Oct 30 19:51:42.526392 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.526370 4990 scope.go:122] "RemoveContainer" containerID="57dfc5624f8e18fe054559c8a716545b7deead63af06fa5b9e0aa28f985543ce" Oct 30 19:51:42.560691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.560661 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:51:42.567476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.567445 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-67df66465f-c89ft"] Oct 30 19:51:42.600168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.599914 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:51:42.604239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:42.604210 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-bbff87ff6-svjc2"] Oct 30 19:51:43.569142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:43.569049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.407023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:44.406975 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" path="/var/lib/kubelet/pods/4bd63e0b-5646-4faf-99cd-a0a93396ffd1/volumes" Oct 30 19:51:44.410116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:44.407841 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" path="/var/lib/kubelet/pods/e04d87cf-c70e-4048-b18e-faeaaa878a2e/volumes" Oct 30 19:51:45.487883 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:45.487843 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:45.637653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:45.637616 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:45.910172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:45.910135 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:47.513673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:47.513603 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:49.177273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:49.177233 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:50.951046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:50.951009 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:52.523822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:52.523777 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:52.757867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:52.757800 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" podUID="88f54105-8a8b-4bd0-8644-9394532154f3" containerName="main" containerID="cri-o://075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a" gracePeriod=45 Oct 30 19:51:53.040705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.040648 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:51:53.184861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.184818 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.184914 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.184964 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\" (UniqueName: \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185003 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185041 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185173 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:53.185356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185217 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\") pod \"88f54105-8a8b-4bd0-8644-9394532154f3\" (UID: \"88f54105-8a8b-4bd0-8644-9394532154f3\") " Oct 30 19:51:53.185478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185419 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:53.185576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.185548 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:53.186025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.186005 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.186202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.186028 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.186202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.186042 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.187498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.187471 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:53.187703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.187674 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:53.187795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.187720 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj" pod "88f54105-8a8b-4bd0-8644-9394532154f3" (UID: "88f54105-8a8b-4bd0-8644-9394532154f3"). InnerVolumeSpecName "kube-api-access-5vsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:53.288418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.288371 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f54105-8a8b-4bd0-8644-9394532154f3-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.288418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.288418 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsvj\" (UniqueName: \"kubernetes.io/projected/88f54105-8a8b-4bd0-8644-9394532154f3-kube-api-access-5vsvj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.288646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.288432 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f54105-8a8b-4bd0-8644-9394532154f3-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:53.393683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.393645 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg"] Oct 30 19:51:53.394999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.394971 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" containerName="payload-processing" Oct 30 19:51:53.395063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395004 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" containerName="payload-processing" Oct 30 19:51:53.395063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395042 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" containerName="payload-pre-processing" Oct 30 19:51:53.395063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395051 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" containerName="payload-pre-processing" Oct 30 19:51:53.395193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395105 4990 cpu_manager.go:414] "containerMap: removing container" podUID="88f54105-8a8b-4bd0-8644-9394532154f3" containerName="main" Oct 30 19:51:53.395193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395114 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f54105-8a8b-4bd0-8644-9394532154f3" containerName="main" Oct 30 19:51:53.395331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395317 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd63e0b-5646-4faf-99cd-a0a93396ffd1" containerName="payload-processing" Oct 30 19:51:53.395367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395341 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04d87cf-c70e-4048-b18e-faeaaa878a2e" containerName="payload-pre-processing" Oct 30 19:51:53.395367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.395356 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f54105-8a8b-4bd0-8644-9394532154f3" containerName="main" Oct 30 19:51:53.400063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.400042 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.412725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.412335 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:53.416419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.416378 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg"] Oct 30 19:51:53.417884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.417865 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:51:53.492337 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.492148 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdx96\" (UniqueName: \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.492558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.492504 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.492558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.492545 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.594460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.594424 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.594951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.594465 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.594951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.594614 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdx96\" (UniqueName: \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.595298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.595274 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.597084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.597059 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.614300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.614269 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdx96\" (UniqueName: \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\") pod \"maas-api-revoke-sub-e514d1f8-vbwbg\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:53.637619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.637582 4990 generic.go:356] "Generic (PLEG): container finished" podID="88f54105-8a8b-4bd0-8644-9394532154f3" containerID="075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a" exitCode=0 Oct 30 19:51:53.637790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.637630 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" Oct 30 19:51:53.637790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.637666 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" event={"ID":"88f54105-8a8b-4bd0-8644-9394532154f3","Type":"ContainerDied","Data":"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a"} Oct 30 19:51:53.637790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.637718 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x" event={"ID":"88f54105-8a8b-4bd0-8644-9394532154f3","Type":"ContainerDied","Data":"812f0aa78fb9f19339d11df137bc499ebed6415f3eff2a3117e4e710583d603b"} Oct 30 19:51:53.637790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.637741 4990 scope.go:122] "RemoveContainer" containerID="075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a" Oct 30 19:51:53.649323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.649303 4990 scope.go:122] "RemoveContainer" containerID="075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a" Oct 30 19:51:53.649642 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:51:53.649623 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a\": container with ID starting with 075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a not found: ID does not exist" containerID="075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a" Oct 30 19:51:53.649699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.649652 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a"} err="failed to get container status \"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a\": rpc error: code = NotFound desc = could not find container \"075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a\": container with ID starting with 075f3ea78b86b2ff014e517a5ab2308db10b2b9bcabb98c78e7ecad93ce9322a not found: ID does not exist" Oct 30 19:51:53.684593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.684549 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:51:53.689198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.689169 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/sub-test-model-d9eb76-kserve-796db989dc-6f44x"] Oct 30 19:51:53.719289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:53.719256 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:54.027945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.027919 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg"] Oct 30 19:51:54.029616 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:51:54.029592 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da5f701_c757_4c29_8abb_cc5fed321176.slice/crio-275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0 WatchSource:0}: Error finding container 275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0: Status 404 returned error can't find the container with id 275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0 Oct 30 19:51:54.156454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.156419 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:54.404760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.404719 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f54105-8a8b-4bd0-8644-9394532154f3" path="/var/lib/kubelet/pods/88f54105-8a8b-4bd0-8644-9394532154f3/volumes" Oct 30 19:51:54.658482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.658390 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" event={"ID":"4da5f701-c757-4c29-8abb-cc5fed321176","Type":"ContainerStarted","Data":"a92d5080a6ffe6e36a39ba8f0691f0dd40a8c4c5fb819b91d53e1673c4828283"} Oct 30 19:51:54.658896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.658492 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" event={"ID":"4da5f701-c757-4c29-8abb-cc5fed321176","Type":"ContainerStarted","Data":"275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0"} Oct 30 19:51:54.696622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:54.696569 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" podStartSLOduration=1.696553995 podStartE2EDuration="1.696553995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:54.691730494 +0000 UTC m=+2822.403255863" watchObservedRunningTime="2026-10-30 19:51:54.696553995 +0000 UTC m=+2822.408079360" Oct 30 19:51:55.674486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:55.674453 4990 generic.go:356] "Generic (PLEG): container finished" podID="4da5f701-c757-4c29-8abb-cc5fed321176" containerID="a92d5080a6ffe6e36a39ba8f0691f0dd40a8c4c5fb819b91d53e1673c4828283" exitCode=0 Oct 30 19:51:55.675021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:55.674573 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" event={"ID":"4da5f701-c757-4c29-8abb-cc5fed321176","Type":"ContainerDied","Data":"a92d5080a6ffe6e36a39ba8f0691f0dd40a8c4c5fb819b91d53e1673c4828283"} Oct 30 19:51:56.034573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:56.034483 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:56.888903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:56.888876 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:57.047545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.047456 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\") pod \"4da5f701-c757-4c29-8abb-cc5fed321176\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " Oct 30 19:51:57.047545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.047520 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\" (UniqueName: \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\") pod \"4da5f701-c757-4c29-8abb-cc5fed321176\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " Oct 30 19:51:57.047732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.047707 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\") pod \"4da5f701-c757-4c29-8abb-cc5fed321176\" (UID: \"4da5f701-c757-4c29-8abb-cc5fed321176\") " Oct 30 19:51:57.047869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.047844 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca" pod "4da5f701-c757-4c29-8abb-cc5fed321176" (UID: "4da5f701-c757-4c29-8abb-cc5fed321176"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:57.048311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.048284 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:57.049859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.049827 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca" pod "4da5f701-c757-4c29-8abb-cc5fed321176" (UID: "4da5f701-c757-4c29-8abb-cc5fed321176"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:57.049941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.049868 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96" pod "4da5f701-c757-4c29-8abb-cc5fed321176" (UID: "4da5f701-c757-4c29-8abb-cc5fed321176"). InnerVolumeSpecName "kube-api-access-xdx96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:57.150347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.150313 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdx96\" (UniqueName: \"kubernetes.io/projected/4da5f701-c757-4c29-8abb-cc5fed321176-kube-api-access-xdx96\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:57.150347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.150343 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5f701-c757-4c29-8abb-cc5fed321176-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:51:57.694932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.694904 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" Oct 30 19:51:57.695140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.694933 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg" event={"ID":"4da5f701-c757-4c29-8abb-cc5fed321176","Type":"ContainerDied","Data":"275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0"} Oct 30 19:51:57.695140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.694982 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275ed3bb96acc90ab892b1e416ed2a6360e134aa84685d5e70184a5e3e892db0" Oct 30 19:51:57.844907 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:57.844716 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:59.601277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:51:59.601234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.329047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:01.329006 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.805349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:01.805302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.180436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:02.180335 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.188746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.188713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.353136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.349195 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.896216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.896183 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5"] Oct 30 19:52:03.896950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.896933 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4da5f701-c757-4c29-8abb-cc5fed321176" containerName="revoke-keys" Oct 30 19:52:03.897005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.896951 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da5f701-c757-4c29-8abb-cc5fed321176" containerName="revoke-keys" Oct 30 19:52:03.897114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.897104 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da5f701-c757-4c29-8abb-cc5fed321176" containerName="revoke-keys" Oct 30 19:52:03.901196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.901176 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:03.918953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.918927 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5"] Oct 30 19:52:03.949418 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.949381 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:03.949613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.949495 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:03.949613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.949561 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95r89\" (UniqueName: \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:03.957468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.957440 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:03.957641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:03.957623 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:52:04.050827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.050796 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.051006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.050879 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.051006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.050920 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95r89\" (UniqueName: \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.051602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.051575 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.053270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.053248 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.075170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.075145 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95r89\" (UniqueName: \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\") pod \"maas-api-revoke-sub-aa33c00c-vbcf5\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.217149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.217037 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:04.522632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.522601 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5"] Oct 30 19:52:04.524191 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:04.524164 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad714d7d_6b6c_48c2_89d2_fd6ea684ab80.slice/crio-94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a WatchSource:0}: Error finding container 94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a: Status 404 returned error can't find the container with id 94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a Oct 30 19:52:04.756261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:04.756222 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" event={"ID":"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80","Type":"ContainerStarted","Data":"94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a"} Oct 30 19:52:05.078274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.078236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.100229 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:52:05.100190 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:52:05.201391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.201316 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:52:05.206035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.206016 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.220004 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.219982 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ed2ede\"" Oct 30 19:52:05.225062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.225041 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:52:05.237976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.237931 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.271888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.271854 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.272048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.271984 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtwn\" (UniqueName: \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.291598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.291565 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ed2ede-dockercfg-vxnt5\"" Oct 30 19:52:05.384543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.384494 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.402171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.401374 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.405214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.405171 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtwn\" (UniqueName: \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.431154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.405416 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:52:05.431154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.422720 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:52:05.431154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.422836 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.455401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.455319 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtwn\" (UniqueName: \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.471643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.471605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.521849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.521811 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652h9\" (UniqueName: \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.522011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.521887 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:05.522582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.522364 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.627150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.627033 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.627315 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.627223 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652h9\" (UniqueName: \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.628226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.628199 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.654429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.654352 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-652h9\" (UniqueName: \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\") pod \"payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.765773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.765680 4990 generic.go:356] "Generic (PLEG): container finished" podID="ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" containerID="d41553fd0ab7e2b1d1eea30d02cc14ca657bf9beca8cd76f485cb5938ccfe176" exitCode=0 Oct 30 19:52:05.765913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.765768 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" event={"ID":"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80","Type":"ContainerDied","Data":"d41553fd0ab7e2b1d1eea30d02cc14ca657bf9beca8cd76f485cb5938ccfe176"} Oct 30 19:52:05.777234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.777206 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:05.846759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:05.846331 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:52:05.850982 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:05.850950 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324d2b9c_673e_4b00_97df_9a168e4d37d9.slice/crio-e655a4389c4a740018b96575741d3200bda2eed020f97dea8a01cbcaee40ec2a WatchSource:0}: Error finding container e655a4389c4a740018b96575741d3200bda2eed020f97dea8a01cbcaee40ec2a: Status 404 returned error can't find the container with id e655a4389c4a740018b96575741d3200bda2eed020f97dea8a01cbcaee40ec2a Oct 30 19:52:06.090735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.090712 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:52:06.093559 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:06.093533 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f94408e_1d81_4171_9385_2b2e933c5ab1.slice/crio-72c2ae3171a2972067853be3f86ab944052b0817f10e78f0013cd1608765534a WatchSource:0}: Error finding container 72c2ae3171a2972067853be3f86ab944052b0817f10e78f0013cd1608765534a: Status 404 returned error can't find the container with id 72c2ae3171a2972067853be3f86ab944052b0817f10e78f0013cd1608765534a Oct 30 19:52:06.577690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.577652 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:52:06.583771 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.583739 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.599730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.597969 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-ec2fe7\"/\"default-dockercfg-9x7f7\"" Oct 30 19:52:06.600877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.599931 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-ec2fe7\"/\"auth-test-model-ec2fe7-kserve-self-signed-certs\"" Oct 30 19:52:06.600877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.600043 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-ec2fe7\"/\"openshift-service-ca.crt\"" Oct 30 19:52:06.601082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.600892 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:52:06.601987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.601964 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-ec2fe7\"/\"kube-root-ca.crt\"" Oct 30 19:52:06.642874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.642820 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.643059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.642890 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.643059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.642987 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.643254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.643183 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.643320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.643283 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dpg\" (UniqueName: \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.643492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.643472 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.683836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.683792 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:06.744849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.744807 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dpg\" (UniqueName: \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745010 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745132 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745182 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745209 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745331 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745502 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745585 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.745725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.745694 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.747820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.747795 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.748048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.748025 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.761929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.761892 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dpg\" (UniqueName: \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\") pod \"auth-test-model-ec2fe7-kserve-64c784bb69-29m2r\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.778375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.778325 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" event={"ID":"324d2b9c-673e-4b00-97df-9a168e4d37d9","Type":"ContainerStarted","Data":"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413"} Oct 30 19:52:06.778524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.778396 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" event={"ID":"324d2b9c-673e-4b00-97df-9a168e4d37d9","Type":"ContainerStarted","Data":"e655a4389c4a740018b96575741d3200bda2eed020f97dea8a01cbcaee40ec2a"} Oct 30 19:52:06.778524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.778469 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:06.780175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.780146 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" event={"ID":"5f94408e-1d81-4171-9385-2b2e933c5ab1","Type":"ContainerStarted","Data":"1b59fdd40a81316b30aa847ed2f6ae864cc3f1d98cb135fe815466eb1f6b0fc8"} Oct 30 19:52:06.780309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.780197 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" event={"ID":"5f94408e-1d81-4171-9385-2b2e933c5ab1","Type":"ContainerStarted","Data":"72c2ae3171a2972067853be3f86ab944052b0817f10e78f0013cd1608765534a"} Oct 30 19:52:06.780309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.780254 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:06.824429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.820348 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" podStartSLOduration=1.8203256890000001 podStartE2EDuration="1.820325689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:06.813368806 +0000 UTC m=+2834.524894173" watchObservedRunningTime="2026-10-30 19:52:06.820325689 +0000 UTC m=+2834.531851059" Oct 30 19:52:06.921469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.905428 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:06.930321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:06.922584 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" podStartSLOduration=1.922565472 podStartE2EDuration="1.922565472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:06.901138392 +0000 UTC m=+2834.612663762" watchObservedRunningTime="2026-10-30 19:52:06.922565472 +0000 UTC m=+2834.634090840" Oct 30 19:52:07.105529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.105501 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:07.120134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.120077 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\") pod \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " Oct 30 19:52:07.120310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.120200 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\" (UniqueName: \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\") pod \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " Oct 30 19:52:07.120470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.120442 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca" pod "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" (UID: "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:07.120558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.120537 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\") pod \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\" (UID: \"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80\") " Oct 30 19:52:07.121248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.121215 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:07.123387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.123356 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca" pod "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" (UID: "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:07.123754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.123733 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89" pod "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" (UID: "ad714d7d-6b6c-48c2-89d2-fd6ea684ab80"). InnerVolumeSpecName "kube-api-access-95r89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:07.222856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.222772 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:07.222856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.222807 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95r89\" (UniqueName: \"kubernetes.io/projected/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80-kube-api-access-95r89\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:07.313086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.313041 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:52:07.329628 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:07.316868 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75985826_9fc3_474e_b98c_9d3874e0ad80.slice/crio-cac3a0068e8d80cd713d762b1c51076264114ca740da4c316e36a152c97396f0 WatchSource:0}: Error finding container cac3a0068e8d80cd713d762b1c51076264114ca740da4c316e36a152c97396f0: Status 404 returned error can't find the container with id cac3a0068e8d80cd713d762b1c51076264114ca740da4c316e36a152c97396f0 Oct 30 19:52:07.340330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.331171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:07.792012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.791975 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" event={"ID":"75985826-9fc3-474e-b98c-9d3874e0ad80","Type":"ContainerStarted","Data":"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a"} Oct 30 19:52:07.792510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.792029 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" event={"ID":"75985826-9fc3-474e-b98c-9d3874e0ad80","Type":"ContainerStarted","Data":"cac3a0068e8d80cd713d762b1c51076264114ca740da4c316e36a152c97396f0"} Oct 30 19:52:07.793613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.793593 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" Oct 30 19:52:07.793720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.793631 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5" event={"ID":"ad714d7d-6b6c-48c2-89d2-fd6ea684ab80","Type":"ContainerDied","Data":"94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a"} Oct 30 19:52:07.793720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.793683 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94125423e3ffe9ddbc6144cef34abcccf3cd57b8eba1cd134e5338dcc4d0ba8a" Oct 30 19:52:07.834378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.834311 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" podStartSLOduration=1.834290417 podStartE2EDuration="1.834290417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:07.826983049 +0000 UTC m=+2835.538508415" watchObservedRunningTime="2026-10-30 19:52:07.834290417 +0000 UTC m=+2835.545815788" Oct 30 19:52:07.912650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:07.912620 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:09.173320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:09.173035 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.352123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.352053 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.494199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.494160 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.722211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.715316 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:52:10.722211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.716588 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" containerName="revoke-keys" Oct 30 19:52:10.722211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.716612 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" containerName="revoke-keys" Oct 30 19:52:10.722211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.716842 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" containerName="revoke-keys" Oct 30 19:52:10.742185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.742154 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:52:10.742554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.742534 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.787371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.787304 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-852709-dockercfg-2rxgs\"" Oct 30 19:52:10.797691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.796350 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-852709\"" Oct 30 19:52:10.797691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.797389 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfkj\" (UniqueName: \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.797691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.797513 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.893183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.893152 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:52:10.909142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.903906 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfkj\" (UniqueName: \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.909142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.904020 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.909142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.905016 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:10.912117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.911210 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:52:10.912117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.911303 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:10.923979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:10.923955 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfkj\" (UniqueName: \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\") pod \"payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:11.005749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.005670 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfnc\" (UniqueName: \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.005950 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.005932 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.107328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.107299 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfnc\" (UniqueName: \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.107494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.107474 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.108202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.108177 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.124512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.124486 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfnc\" (UniqueName: \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\") pod \"payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.188803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.188761 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:11.243429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.243400 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.419465 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.418897 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:11.496370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.496337 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:52:11.500731 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:11.500698 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5cdadf_9fda_4c36_8843_dceacba7a5cd.slice/crio-d7e331469d0624177730cf79e549c7319b1973ff2d466bb02b2e3732133dc125 WatchSource:0}: Error finding container d7e331469d0624177730cf79e549c7319b1973ff2d466bb02b2e3732133dc125: Status 404 returned error can't find the container with id d7e331469d0624177730cf79e549c7319b1973ff2d466bb02b2e3732133dc125 Oct 30 19:52:11.590759 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:11.590734 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2dd0ae_31f2_46b1_a90d_2e82e024178b.slice/crio-65a71b60f7d25335d0829258c930f5d2277f36c81fed88a9a69b0b2b077f0266 WatchSource:0}: Error finding container 65a71b60f7d25335d0829258c930f5d2277f36c81fed88a9a69b0b2b077f0266: Status 404 returned error can't find the container with id 65a71b60f7d25335d0829258c930f5d2277f36c81fed88a9a69b0b2b077f0266 Oct 30 19:52:11.590998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.590976 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:52:11.903174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.903138 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" event={"ID":"7d2dd0ae-31f2-46b1-a90d-2e82e024178b","Type":"ContainerStarted","Data":"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573"} Oct 30 19:52:11.903485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.903443 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" event={"ID":"7d2dd0ae-31f2-46b1-a90d-2e82e024178b","Type":"ContainerStarted","Data":"65a71b60f7d25335d0829258c930f5d2277f36c81fed88a9a69b0b2b077f0266"} Oct 30 19:52:11.903671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.903648 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:11.905390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.905362 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" event={"ID":"ca5cdadf-9fda-4c36-8843-dceacba7a5cd","Type":"ContainerStarted","Data":"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c"} Oct 30 19:52:11.905528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.905411 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" event={"ID":"ca5cdadf-9fda-4c36-8843-dceacba7a5cd","Type":"ContainerStarted","Data":"d7e331469d0624177730cf79e549c7319b1973ff2d466bb02b2e3732133dc125"} Oct 30 19:52:11.905528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.905484 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:11.941639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.941586 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" podStartSLOduration=1.941570005 podStartE2EDuration="1.941570005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:11.938236143 +0000 UTC m=+2839.649761510" watchObservedRunningTime="2026-10-30 19:52:11.941570005 +0000 UTC m=+2839.653095380" Oct 30 19:52:11.976396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:11.976344 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" podStartSLOduration=1.9763285769999999 podStartE2EDuration="1.976328577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:11.971936746 +0000 UTC m=+2839.683462113" watchObservedRunningTime="2026-10-30 19:52:11.976328577 +0000 UTC m=+2839.687853944" Oct 30 19:52:13.125118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:13.124611 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.777139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:14.775337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.183527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:16.183488 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.515472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:16.515371 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.912637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:16.912605 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:16.926267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:16.926241 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:17.003024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:17.002992 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:52:17.797661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:17.797627 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:52:17.798200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:17.797713 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:52:18.016588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:18.015786 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:19.712016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:19.711980 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.536180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:21.527488 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.245644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.245610 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:52:22.257825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.257774 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.272769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.272731 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"sub-test-model-870193-kserve-self-signed-certs\"" Oct 30 19:52:22.281314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.281253 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"default-dockercfg-pfhbn\"" Oct 30 19:52:22.283473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.283442 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"kube-root-ca.crt\"" Oct 30 19:52:22.284925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.284901 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"openshift-service-ca.crt\"" Oct 30 19:52:22.288247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.288216 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:52:22.341634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.341593 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.397154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397086 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.397376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397203 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2w6k\" (UniqueName: \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.397376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397240 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.397505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397442 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.397587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397568 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.397667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.397609 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502154 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502233 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502449 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502502 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2w6k\" (UniqueName: \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502544 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.502732 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.511878 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.512694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.512232 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.514351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.514311 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.516155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.515289 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.519278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.519225 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.535070 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.534704 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2w6k\" (UniqueName: \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\") pod \"sub-test-model-870193-kserve-6599d7b4f-lrcmn\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.588116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.587734 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:22.918704 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:22.918667 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0856eda_3688_40b3_9e1c_b0bc2ae5ab21.slice/crio-ca5952af3d5bc692af3d8060e5fe90d27365bd20019f6d4c0c5caf0c991e8ad0 WatchSource:0}: Error finding container ca5952af3d5bc692af3d8060e5fe90d27365bd20019f6d4c0c5caf0c991e8ad0: Status 404 returned error can't find the container with id ca5952af3d5bc692af3d8060e5fe90d27365bd20019f6d4c0c5caf0c991e8ad0 Oct 30 19:52:22.919301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.919278 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:52:22.948017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.947985 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:52:22.948905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:22.948883 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:52:23.049366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:23.049328 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" event={"ID":"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21","Type":"ContainerStarted","Data":"cdef5cd5b77789d66705fdaabeff46569dd7db79685c1d67e9a4deb770f051aa"} Oct 30 19:52:23.049560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:23.049375 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" event={"ID":"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21","Type":"ContainerStarted","Data":"ca5952af3d5bc692af3d8060e5fe90d27365bd20019f6d4c0c5caf0c991e8ad0"} Oct 30 19:52:23.091113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:23.091030 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" podStartSLOduration=1.090997746 podStartE2EDuration="1.090997746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:23.0873222 +0000 UTC m=+2850.798847567" watchObservedRunningTime="2026-10-30 19:52:23.090997746 +0000 UTC m=+2850.802523113" Oct 30 19:52:23.164361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:23.164308 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:23.588542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:23.588507 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:24.856185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:24.854263 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:27.009115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:27.003236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:29.296075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:29.296025 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:30.945526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:30.945481 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.455906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:32.455867 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.588588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:32.588549 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:32.607735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:32.607668 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:32.843461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:32.841783 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:33.234886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.234793 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:52:33.315074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.315019 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:52:33.321217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.321182 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.333853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.333825 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5250ee\"/\"default-dockercfg-nrcl7\"" Oct 30 19:52:33.334956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.334932 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-5250ee\"/\"auth-test-model-5250ee-kserve-self-signed-certs\"" Oct 30 19:52:33.335165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.334993 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5250ee\"/\"openshift-service-ca.crt\"" Oct 30 19:52:33.338931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.338891 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:52:33.339297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.339045 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-5250ee\"/\"kube-root-ca.crt\"" Oct 30 19:52:33.385537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.383689 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:33.403697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.403658 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.403879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.403716 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.403879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.403844 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.404016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.403891 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg58m\" (UniqueName: \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.404169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.404138 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.404312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.404280 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505403 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505531 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505586 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505626 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505693 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505757 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg58m\" (UniqueName: \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.505994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.505918 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.506362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.506203 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.506585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.506555 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.509131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.508773 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.509131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.508896 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.558964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.558606 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg58m\" (UniqueName: \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\") pod \"auth-test-model-5250ee-kserve-67c657f985-4597w\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.646502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.646465 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:33.984403 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:33.984364 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:52:34.006378 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:34.006337 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eecc7cd_7a8c_46bf_acf3_7014cafbb2e5.slice/crio-085fa65ce9a1a271bc75fa3cc613591ae387bd096b14cd36a87a5a1f6e6c6c5c WatchSource:0}: Error finding container 085fa65ce9a1a271bc75fa3cc613591ae387bd096b14cd36a87a5a1f6e6c6c5c: Status 404 returned error can't find the container with id 085fa65ce9a1a271bc75fa3cc613591ae387bd096b14cd36a87a5a1f6e6c6c5c Oct 30 19:52:34.234468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:34.234363 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" event={"ID":"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5","Type":"ContainerStarted","Data":"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47"} Oct 30 19:52:34.234468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:34.234416 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" event={"ID":"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5","Type":"ContainerStarted","Data":"085fa65ce9a1a271bc75fa3cc613591ae387bd096b14cd36a87a5a1f6e6c6c5c"} Oct 30 19:52:34.274027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:34.273943 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" podStartSLOduration=1.273924399 podStartE2EDuration="1.273924399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:34.269555678 +0000 UTC m=+2861.981081042" watchObservedRunningTime="2026-10-30 19:52:34.273924399 +0000 UTC m=+2861.985449765" Oct 30 19:52:34.426963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:34.426920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.647022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:34.646982 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:35.105617 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:52:35.105577 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:52:36.200313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.200206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.254885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.254849 4990 generic.go:356] "Generic (PLEG): container finished" podID="2dba8c98-73c7-4670-b719-7304469946ca" containerID="263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960" exitCode=0 Oct 30 19:52:36.255074 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.254923 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerDied","Data":"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960"} Oct 30 19:52:36.255440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.255422 4990 scope.go:122] "RemoveContainer" containerID="263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960" Oct 30 19:52:36.257268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.257237 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:36.257356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.257307 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:36.257356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:36.257347 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:37.281114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:37.274225 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerStarted","Data":"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4"} Oct 30 19:52:37.755012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:37.754958 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.096119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.095502 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:52:38.096119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.095672 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:52:38.096119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.095729 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 19:52:38.616057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.616011 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s"] Oct 30 19:52:38.623008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.622973 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.640125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.640069 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s"] Oct 30 19:52:38.642932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.642903 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:38.643147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.643031 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:52:38.724501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.724457 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7wk\" (UniqueName: \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.724719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.724672 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.724788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.724715 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.825972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.825924 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7wk\" (UniqueName: \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.826205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.826143 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.826205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.826174 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.826788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.826763 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.828681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.828654 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.843346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.843316 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7wk\" (UniqueName: \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\") pod \"maas-api-revoke-keys-e2e-ipp-a-54086b-q967s\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:38.945944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:38.945839 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:39.117220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:39.116768 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:52:39.117220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:39.116846 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 19:52:39.303285 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:39.303079 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd45a88_f5f4_40c4_950c_069529bb4b17.slice/crio-78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a WatchSource:0}: Error finding container 78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a: Status 404 returned error can't find the container with id 78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a Oct 30 19:52:39.306581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:39.306544 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s"] Oct 30 19:52:39.608858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:39.608818 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.102150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.102109 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:52:40.102560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.102174 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 19:52:40.323384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.323349 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" event={"ID":"abd45a88-f5f4-40c4-950c-069529bb4b17","Type":"ContainerStarted","Data":"afa31ac9be5d4d2ba96154cd15154471d2c13ea41590205e2645d60ecb3a63a5"} Oct 30 19:52:40.323565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.323406 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" event={"ID":"abd45a88-f5f4-40c4-950c-069529bb4b17","Type":"ContainerStarted","Data":"78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a"} Oct 30 19:52:40.362512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.362401 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" podStartSLOduration=2.362387001 podStartE2EDuration="2.362387001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:40.356615932 +0000 UTC m=+2868.068141298" watchObservedRunningTime="2026-10-30 19:52:40.362387001 +0000 UTC m=+2868.073912361" Oct 30 19:52:40.604175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.602989 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.740311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:40.740174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.097119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:41.096724 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:52:41.097119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:41.096787 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 19:52:41.303970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:41.303930 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.339563 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:41.339528 4990 generic.go:356] "Generic (PLEG): container finished" podID="abd45a88-f5f4-40c4-950c-069529bb4b17" containerID="afa31ac9be5d4d2ba96154cd15154471d2c13ea41590205e2645d60ecb3a63a5" exitCode=0 Oct 30 19:52:41.339742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:41.339616 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" event={"ID":"abd45a88-f5f4-40c4-950c-069529bb4b17","Type":"ContainerDied","Data":"afa31ac9be5d4d2ba96154cd15154471d2c13ea41590205e2645d60ecb3a63a5"} Oct 30 19:52:42.093578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.093539 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:52:42.094030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.094000 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 19:52:42.557818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.557783 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:42.604595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.604506 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\") pod \"abd45a88-f5f4-40c4-950c-069529bb4b17\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " Oct 30 19:52:42.604798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.604776 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\") pod \"abd45a88-f5f4-40c4-950c-069529bb4b17\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " Oct 30 19:52:42.604892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.604841 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\" (UniqueName: \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\") pod \"abd45a88-f5f4-40c4-950c-069529bb4b17\" (UID: \"abd45a88-f5f4-40c4-950c-069529bb4b17\") " Oct 30 19:52:42.606735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.606707 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca" pod "abd45a88-f5f4-40c4-950c-069529bb4b17" (UID: "abd45a88-f5f4-40c4-950c-069529bb4b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:42.609464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.609436 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk" pod "abd45a88-f5f4-40c4-950c-069529bb4b17" (UID: "abd45a88-f5f4-40c4-950c-069529bb4b17"). InnerVolumeSpecName "kube-api-access-8d7wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:42.609679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.609654 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca" pod "abd45a88-f5f4-40c4-950c-069529bb4b17" (UID: "abd45a88-f5f4-40c4-950c-069529bb4b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:42.707544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.707501 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:42.707544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.707543 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7wk\" (UniqueName: \"kubernetes.io/projected/abd45a88-f5f4-40c4-950c-069529bb4b17-kube-api-access-8d7wk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:42.707817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:42.707557 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd45a88-f5f4-40c4-950c-069529bb4b17-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:52:43.100035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.100004 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:52:43.100435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.100412 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:52:43.101316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.101296 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:52:43.219081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.219047 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:43.358299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.358220 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" Oct 30 19:52:43.358299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.358253 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s" event={"ID":"abd45a88-f5f4-40c4-950c-069529bb4b17","Type":"ContainerDied","Data":"78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a"} Oct 30 19:52:43.358489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.358305 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78eb8712a5fa3b6505b837cc41f4d4b4bda6f7ef9e1810b68185a7228752c81a" Oct 30 19:52:43.647622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.647546 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:43.660347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:43.660319 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:44.379880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:44.379845 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:52:44.774630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:44.774533 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.314461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:46.314424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.459431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:46.459385 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.594311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:46.594210 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:48.107429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:48.107389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:49.727535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:49.727497 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:51.387546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:51.385137 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.814820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.814784 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 19:52:52.815612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.815591 4990 cpu_manager.go:414] "containerMap: removing container" podUID="abd45a88-f5f4-40c4-950c-069529bb4b17" containerName="revoke-keys" Oct 30 19:52:52.815612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.815608 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd45a88-f5f4-40c4-950c-069529bb4b17" containerName="revoke-keys" Oct 30 19:52:52.815770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.815758 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd45a88-f5f4-40c4-950c-069529bb4b17" containerName="revoke-keys" Oct 30 19:52:52.820265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.820241 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.832843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.832815 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cd4bde-openshift-default-dockercfg-dhs49\"" Oct 30 19:52:52.849570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.849533 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 19:52:52.882395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882357 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882404 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nft9t\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882497 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882608 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882665 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882698 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882766 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.882876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882792 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.883050 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.882899 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.983992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.983938 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984015 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984047 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984147 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984176 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984297 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984481 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984504 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984538 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984549 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nft9t\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984646 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984828 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.984972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.984942 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.985159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.985139 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.986863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.986828 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:52.986973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:52.986829 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:53.001729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.001698 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:53.001917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.001882 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nft9t\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:53.147558 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.147405 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:53.152333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.151431 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cd4bde-openshift-default-dockercfg-dhs49\"" Oct 30 19:52:53.159129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.157121 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:53.502007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.501972 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 19:52:53.503610 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:52:53.503580 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523c919e_98f3_4b9d_8199_73d2d9ae649e.slice/crio-db9ef7b7addb96630d16eec7a7b0163cb60640a0bcce2c3b2b6bfebbfab42e6d WatchSource:0}: Error finding container db9ef7b7addb96630d16eec7a7b0163cb60640a0bcce2c3b2b6bfebbfab42e6d: Status 404 returned error can't find the container with id db9ef7b7addb96630d16eec7a7b0163cb60640a0bcce2c3b2b6bfebbfab42e6d Oct 30 19:52:53.506510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.506475 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:53.506611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.506575 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:53.506684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:53.506623 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:52:54.533116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:54.523703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.561156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:54.549798 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" event={"ID":"523c919e-98f3-4b9d-8199-73d2d9ae649e","Type":"ContainerStarted","Data":"a4aad552597dee35ca2a94c30c2d30a076f11fc985f1a9e378459ebd7b889f36"} Oct 30 19:52:54.561156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:54.549870 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" event={"ID":"523c919e-98f3-4b9d-8199-73d2d9ae649e","Type":"ContainerStarted","Data":"db9ef7b7addb96630d16eec7a7b0163cb60640a0bcce2c3b2b6bfebbfab42e6d"} Oct 30 19:52:54.633609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:54.633562 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podStartSLOduration=2.6335476 podStartE2EDuration="2.6335476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:54.622989893 +0000 UTC m=+2882.334515264" watchObservedRunningTime="2026-10-30 19:52:54.6335476 +0000 UTC m=+2882.345072969" Oct 30 19:52:55.098118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:55.097120 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.158193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:55.158163 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:52:55.159999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:55.159966 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 19:52:55.160141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:55.160025 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 19:52:55.618282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:55.618238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:56.158664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:56.158631 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 19:52:56.158857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:56.158694 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 19:52:57.158557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:57.158513 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 19:52:57.158954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:57.158575 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 19:52:57.252981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:57.251158 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.158512 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:58.158474 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 19:52:58.158695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:58.158533 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 19:52:59.039328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:59.039280 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.158451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:59.158417 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Oct 30 19:52:59.158630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:52:59.158472 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Oct 30 19:53:00.163184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:00.163148 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:53:00.163664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:00.163509 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:53:00.164285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:00.164266 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 19:53:00.618135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:00.618080 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:01.899256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.899215 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:53:01.908370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.908343 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.923372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.923340 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:53:01.927043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.925958 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-5a0a17\"/\"default-dockercfg-ldmrw\"" Oct 30 19:53:01.927043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.926154 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-5a0a17\"/\"kube-root-ca.crt\"" Oct 30 19:53:01.932633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.932607 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-5a0a17\"/\"auth-test-model-5a0a17-kserve-self-signed-certs\"" Oct 30 19:53:01.932785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.932613 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-5a0a17\"/\"openshift-service-ca.crt\"" Oct 30 19:53:01.934237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.934214 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:01.965762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.965723 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8fq\" (UniqueName: \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.965974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.965911 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.965974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.965968 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.966185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.966162 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.966287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.966270 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:01.966335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:01.966322 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068461 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068560 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068609 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068642 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8fq\" (UniqueName: \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068768 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068830 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.068991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.068956 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.069138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.069075 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.069453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.069429 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.071372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.071345 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.071467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.071369 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.076596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.076570 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.085417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.085395 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8fq\" (UniqueName: \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\") pod \"auth-test-model-5a0a17-kserve-6b6c8df44f-srr22\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.221139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.221024 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.233529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.233495 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:02.380115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.380043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.566131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.566081 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:53:02.582990 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:53:02.582961 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2677cbc_5b99_472a_8f84_d36800493e34.slice/crio-a9c37ef4a611ea42ad08b942d17831c4db857ec5e55d9c02574dd481f8de3650 WatchSource:0}: Error finding container a9c37ef4a611ea42ad08b942d17831c4db857ec5e55d9c02574dd481f8de3650: Status 404 returned error can't find the container with id a9c37ef4a611ea42ad08b942d17831c4db857ec5e55d9c02574dd481f8de3650 Oct 30 19:53:02.654847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:02.654809 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" event={"ID":"b2677cbc-5b99-472a-8f84-d36800493e34","Type":"ContainerStarted","Data":"a9c37ef4a611ea42ad08b942d17831c4db857ec5e55d9c02574dd481f8de3650"} Oct 30 19:53:03.665746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:03.665704 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" event={"ID":"b2677cbc-5b99-472a-8f84-d36800493e34","Type":"ContainerStarted","Data":"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018"} Oct 30 19:53:03.708650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:03.708587 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" podStartSLOduration=2.708573614 podStartE2EDuration="2.708573614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:03.704000696 +0000 UTC m=+2891.415526063" watchObservedRunningTime="2026-10-30 19:53:03.708573614 +0000 UTC m=+2891.420098980" Oct 30 19:53:04.000270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:04.000174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.234199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:04.234172 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:05.099341 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:05.099301 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:53:05.606116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:05.605195 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.774115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:05.771925 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:07.262378 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:07.262346 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.889235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:08.889197 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.046049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:09.046008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.195758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:09.195663 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.601413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.601376 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:53:10.602166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.602134 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" podUID="51006b93-2179-4679-a433-fedbac4e8f16" containerName="maas-api" containerID="cri-o://fd993f1e551098c18fb6ef7273a764785de87b6e36bc334f6cc7c5fb64f46cba" gracePeriod=30 Oct 30 19:53:10.602451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.602426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.693310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.693274 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" podUID="51006b93-2179-4679-a433-fedbac4e8f16" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.219:8443/health\": dial tcp 10.134.0.219:8443: connect: connection refused" Oct 30 19:53:10.738956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.738924 4990 generic.go:356] "Generic (PLEG): container finished" podID="51006b93-2179-4679-a433-fedbac4e8f16" containerID="fd993f1e551098c18fb6ef7273a764785de87b6e36bc334f6cc7c5fb64f46cba" exitCode=0 Oct 30 19:53:10.739177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.739005 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" event={"ID":"51006b93-2179-4679-a433-fedbac4e8f16","Type":"ContainerDied","Data":"fd993f1e551098c18fb6ef7273a764785de87b6e36bc334f6cc7c5fb64f46cba"} Oct 30 19:53:10.778701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.778600 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.797732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.797700 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:53:10.798069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.798044 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" podUID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" containerName="payload-pre-processing" containerID="cri-o://8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8" gracePeriod=30 Oct 30 19:53:10.940263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.940071 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.998148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:10.997272 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:53:11.084160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.084047 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.085787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.085761 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\") pod \"51006b93-2179-4679-a433-fedbac4e8f16\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " Oct 30 19:53:11.086116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.085941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\") pod \"51006b93-2179-4679-a433-fedbac4e8f16\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " Oct 30 19:53:11.086116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.085984 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\" (UniqueName: \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\") pod \"51006b93-2179-4679-a433-fedbac4e8f16\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " Oct 30 19:53:11.086116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.086064 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\") pod \"51006b93-2179-4679-a433-fedbac4e8f16\" (UID: \"51006b93-2179-4679-a433-fedbac4e8f16\") " Oct 30 19:53:11.086833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.086338 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca" pod "51006b93-2179-4679-a433-fedbac4e8f16" (UID: "51006b93-2179-4679-a433-fedbac4e8f16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:11.087681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.087363 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51006b93-2179-4679-a433-fedbac4e8f16-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.095179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.094473 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h" pod "51006b93-2179-4679-a433-fedbac4e8f16" (UID: "51006b93-2179-4679-a433-fedbac4e8f16"). InnerVolumeSpecName "kube-api-access-hh26h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:11.095179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.094722 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs" pod "51006b93-2179-4679-a433-fedbac4e8f16" (UID: "51006b93-2179-4679-a433-fedbac4e8f16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:11.111116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.100542 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls" pod "51006b93-2179-4679-a433-fedbac4e8f16" (UID: "51006b93-2179-4679-a433-fedbac4e8f16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:11.140764 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.140737 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:53:11.341288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.339941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\" (UniqueName: \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\") pod \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " Oct 30 19:53:11.341288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.340081 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\") pod \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\" (UID: \"7b2e1510-0528-4d5f-a9e0-a9924e46206a\") " Oct 30 19:53:11.341288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.341064 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.341288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.341110 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh26h\" (UniqueName: \"kubernetes.io/projected/51006b93-2179-4679-a433-fedbac4e8f16-kube-api-access-hh26h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.341288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.341127 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51006b93-2179-4679-a433-fedbac4e8f16-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.382744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.347736 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf" pod "7b2e1510-0528-4d5f-a9e0-a9924e46206a" (UID: "7b2e1510-0528-4d5f-a9e0-a9924e46206a"). InnerVolumeSpecName "kube-api-access-9pwjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:11.382744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.348570 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume" pod "7b2e1510-0528-4d5f-a9e0-a9924e46206a" (UID: "7b2e1510-0528-4d5f-a9e0-a9924e46206a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:11.382744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.364683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.443286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.443253 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwjf\" (UniqueName: \"kubernetes.io/projected/7b2e1510-0528-4d5f-a9e0-a9924e46206a-kube-api-access-9pwjf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.443286 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.443283 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2e1510-0528-4d5f-a9e0-a9924e46206a-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.582354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.582323 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s"] Oct 30 19:53:11.590289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.590255 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54086b-q967s"] Oct 30 19:53:11.601138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.601020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.751034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.750984 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.752062 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.752034 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" Oct 30 19:53:11.752230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.752033 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5" event={"ID":"51006b93-2179-4679-a433-fedbac4e8f16","Type":"ContainerDied","Data":"19383aec75303d1703b931b54571db9fabd6472118d657a45a7aea9fb14ad00c"} Oct 30 19:53:11.752230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.752200 4990 scope.go:122] "RemoveContainer" containerID="fd993f1e551098c18fb6ef7273a764785de87b6e36bc334f6cc7c5fb64f46cba" Oct 30 19:53:11.756324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.756295 4990 generic.go:356] "Generic (PLEG): container finished" podID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" containerID="8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8" exitCode=0 Oct 30 19:53:11.756445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.756428 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" event={"ID":"7b2e1510-0528-4d5f-a9e0-a9924e46206a","Type":"ContainerDied","Data":"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8"} Oct 30 19:53:11.756497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.756477 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" event={"ID":"7b2e1510-0528-4d5f-a9e0-a9924e46206a","Type":"ContainerDied","Data":"d185f4c4742ba385e609c3ff2b73e90c84b6ce5b4f0ff73d0b5edba8a1fcf90d"} Oct 30 19:53:11.756567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.756554 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k" Oct 30 19:53:11.775193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.774769 4990 scope.go:122] "RemoveContainer" containerID="8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8" Oct 30 19:53:11.807719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.807695 4990 scope.go:122] "RemoveContainer" containerID="8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8" Oct 30 19:53:11.808239 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:11.808197 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8\": container with ID starting with 8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8 not found: ID does not exist" containerID="8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8" Oct 30 19:53:11.808350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.808242 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8"} err="failed to get container status \"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8\": rpc error: code = NotFound desc = could not find container \"8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8\": container with ID starting with 8b64c7ce32e616d37a59e692ea001de5b44e19f99c75b69bff15894758fc1ae8 not found: ID does not exist" Oct 30 19:53:11.823714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.823682 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:53:11.830247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.830219 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54086b-7f4d68b79f-vb92k"] Oct 30 19:53:11.863188 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.858079 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:53:11.865169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:11.865144 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54086b-68dd45f94d-cfsk5"] Oct 30 19:53:12.234250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.234147 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:12.249690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.249664 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:12.502163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.494764 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51006b93-2179-4679-a433-fedbac4e8f16" path="/var/lib/kubelet/pods/51006b93-2179-4679-a433-fedbac4e8f16/volumes" Oct 30 19:53:12.502163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.497712 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" path="/var/lib/kubelet/pods/7b2e1510-0528-4d5f-a9e0-a9924e46206a/volumes" Oct 30 19:53:12.502163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.499046 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd45a88-f5f4-40c4-950c-069529bb4b17" path="/var/lib/kubelet/pods/abd45a88-f5f4-40c4-950c-069529bb4b17/volumes" Oct 30 19:53:12.831311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:12.826851 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:53:14.285707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:14.276915 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:15.997936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:15.997635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.685177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.685132 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.735917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.735877 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9235-f174a8"] Oct 30 19:53:17.737081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737054 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" containerName="payload-pre-processing" Oct 30 19:53:17.737081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737078 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" containerName="payload-pre-processing" Oct 30 19:53:17.737342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737152 4990 cpu_manager.go:414] "containerMap: removing container" podUID="51006b93-2179-4679-a433-fedbac4e8f16" containerName="maas-api" Oct 30 19:53:17.737342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737159 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51006b93-2179-4679-a433-fedbac4e8f16" containerName="maas-api" Oct 30 19:53:17.737342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737280 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="51006b93-2179-4679-a433-fedbac4e8f16" containerName="maas-api" Oct 30 19:53:17.737342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.737298 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2e1510-0528-4d5f-a9e0-a9924e46206a" containerName="payload-pre-processing" Oct 30 19:53:17.743128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.743107 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:17.754478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.754433 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9235-f174a8"] Oct 30 19:53:17.758408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.758388 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:53:17.823585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.821145 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlckw\" (UniqueName: \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\") pod \"test-curl-9235-f174a8\" (UID: \"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599\") " pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:17.922808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.922764 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlckw\" (UniqueName: \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\") pod \"test-curl-9235-f174a8\" (UID: \"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599\") " pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:17.940344 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:17.940262 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlckw\" (UniqueName: \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\") pod \"test-curl-9235-f174a8\" (UID: \"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599\") " pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:18.033253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.033221 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2"] Oct 30 19:53:18.037960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.037932 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498ea864-7rmb2"] Oct 30 19:53:18.063567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.063543 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:18.409635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.409602 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25160e8-56c3-4ca9-b6c1-bb5311e134b8" path="/var/lib/kubelet/pods/f25160e8-56c3-4ca9-b6c1-bb5311e134b8/volumes" Oct 30 19:53:18.410108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.410069 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9235-f174a8"] Oct 30 19:53:18.410912 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:53:18.410884 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6dfbef8_cfbf_4f66_ba4f_a794d69eb599.slice/crio-2cc6e4de15852f182e43f579e6645e9d89657a324700ede2f2cb5e5232bbc295 WatchSource:0}: Error finding container 2cc6e4de15852f182e43f579e6645e9d89657a324700ede2f2cb5e5232bbc295: Status 404 returned error can't find the container with id 2cc6e4de15852f182e43f579e6645e9d89657a324700ede2f2cb5e5232bbc295 Oct 30 19:53:18.906113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.902760 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9235-f174a8" event={"ID":"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599","Type":"ContainerStarted","Data":"eefab0029b50c5f48aebecd64d4fa8681a10af664a3ce366b90afa423781bded"} Oct 30 19:53:18.906113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.902813 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9235-f174a8" event={"ID":"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599","Type":"ContainerStarted","Data":"2cc6e4de15852f182e43f579e6645e9d89657a324700ede2f2cb5e5232bbc295"} Oct 30 19:53:18.941648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:18.941591 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9235-f174a8" podStartSLOduration=1.941574353 podStartE2EDuration="1.941574353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:18.936533899 +0000 UTC m=+2906.648059267" watchObservedRunningTime="2026-10-30 19:53:18.941574353 +0000 UTC m=+2906.653099718" Oct 30 19:53:19.337660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:19.337625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:21.007739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:21.007698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.661397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:22.661342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:23.951297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.951203 4990 generic.go:356] "Generic (PLEG): container finished" podID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerID="12755a23009f83152b1610371f670824badf48a24b7ea796e9eb442c67a73ea8" exitCode=0 Oct 30 19:53:23.951297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.951276 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerDied","Data":"12755a23009f83152b1610371f670824badf48a24b7ea796e9eb442c67a73ea8"} Oct 30 19:53:23.951792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.951688 4990 scope.go:122] "RemoveContainer" containerID="12755a23009f83152b1610371f670824badf48a24b7ea796e9eb442c67a73ea8" Oct 30 19:53:23.953481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.953450 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:23.953552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.953523 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:23.953597 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:23.953558 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:24.436743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:24.436692 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.973908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:24.973867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerStarted","Data":"c71b3c642a7917ebb08f2f6bfc934104ffe51aaa93a3c04bd19733a2bda28e48"} Oct 30 19:53:25.172184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:25.172150 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:53:25.193114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:25.192211 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:25.193114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:25.192280 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:26.002129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:25.993724 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.165196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:26.165154 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:26.165405 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:26.165221 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:26.506739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:26.506698 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:53:27.179117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:27.176404 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:27.179117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:27.176473 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:27.611664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:27.611621 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:28.167227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:28.167193 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:28.167535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:28.167503 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:29.172642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:29.172189 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:29.172642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:29.172260 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:29.424921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:29.424830 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:30.178714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:30.178680 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 19:53:30.179478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:30.179444 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 19:53:31.195763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:31.195670 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.239288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:31.239255 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:53:31.240588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:31.240519 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 19:53:31.651807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:31.651767 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.083926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:32.083887 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.087714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:33.087671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:34.103826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.103790 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9235-f174a8"] Oct 30 19:53:34.104397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.104019 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9235-f174a8" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" containerName="test-curl-9235-f174a8" containerID="cri-o://eefab0029b50c5f48aebecd64d4fa8681a10af664a3ce366b90afa423781bded" gracePeriod=2 Oct 30 19:53:34.110133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.110105 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9235-f174a8"] Oct 30 19:53:34.115166 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:34.115130 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9235-f174a8\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:34.337118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.337058 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk"] Oct 30 19:53:34.337884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.337866 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" containerName="test-curl-9235-f174a8" Oct 30 19:53:34.337957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.337888 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" containerName="test-curl-9235-f174a8" Oct 30 19:53:34.338057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.338046 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" containerName="test-curl-9235-f174a8" Oct 30 19:53:34.340774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.340758 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.351983 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:34.351956 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9235-f174a8\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:34.354413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.354359 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:34.354521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.354423 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:53:34.360567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.360543 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk"] Oct 30 19:53:34.529055 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.529021 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkm57\" (UniqueName: \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.529256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.529167 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.529365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.529342 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.631455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.631365 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkm57\" (UniqueName: \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.631455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.631445 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.631712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.631538 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.632217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.632195 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.633860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.633828 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.658530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.658504 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkm57\" (UniqueName: \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\") pod \"maas-api-revoke-sub-a885d7db-cvlhk\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:34.778497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.778452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:34.954820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:34.954747 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:35.116541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.116503 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:53:35.122840 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:35.122771 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:53:35.151069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.151040 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:53:35.272984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.272913 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk"] Oct 30 19:53:35.293882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.293820 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.519876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.519574 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.937155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.923654 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:53:35.937155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.931718 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.950120 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.945951 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5250ee-openshift-default-dockercfg-bzstj\"" Oct 30 19:53:35.952751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.952715 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.964982 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965245 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965461 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965528 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965578 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965659 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsspx\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965693 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965727 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:35.968117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:35.965755 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.066891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.066857 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsspx\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.066907 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.066945 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.066969 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.067123 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067387 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.067209 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.067410 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.067480 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.067567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.067536 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.068326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.068296 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.068493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.068464 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.069058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.068739 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.069058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.069021 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.069294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.069270 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.071323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.071300 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.073701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.073657 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.086802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.086772 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsspx\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.089975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.089925 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.218192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.214873 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" event={"ID":"2b0618b6-428f-41d5-8c70-161b161ebfe4","Type":"ContainerStarted","Data":"3e399028075b49338d93a4b754a77d462440ea97b027cc638052be07535141a5"} Oct 30 19:53:36.218192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.214938 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" event={"ID":"2b0618b6-428f-41d5-8c70-161b161ebfe4","Type":"ContainerStarted","Data":"c918f691441af6619742e84713577787d3b9aa012a8656ce9904773e217654c6"} Oct 30 19:53:36.218192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.217172 4990 generic.go:356] "Generic (PLEG): container finished" podID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" containerID="eefab0029b50c5f48aebecd64d4fa8681a10af664a3ce366b90afa423781bded" exitCode=137 Oct 30 19:53:36.253958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.253576 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" podStartSLOduration=2.253559165 podStartE2EDuration="2.253559165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:36.251463515 +0000 UTC m=+2923.962988881" watchObservedRunningTime="2026-10-30 19:53:36.253559165 +0000 UTC m=+2923.965084531" Oct 30 19:53:36.264509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.264478 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:36.410885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.410618 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:36.497530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.497441 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\" (UniqueName: \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\") pod \"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599\" (UID: \"a6dfbef8-cfbf-4f66-ba4f-a794d69eb599\") " Oct 30 19:53:36.499791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.499759 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw" pod "a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" (UID: "a6dfbef8-cfbf-4f66-ba4f-a794d69eb599"). InnerVolumeSpecName "kube-api-access-dlckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:36.600939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.600904 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlckw\" (UniqueName: \"kubernetes.io/projected/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599-kube-api-access-dlckw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:36.638137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.638104 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:53:36.646677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.646628 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:36.646801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.646715 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:36.646801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.646756 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:53:36.695918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:36.673908 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:37.229460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.229422 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" event={"ID":"b8cbf048-c3a9-40b3-b589-5844ba2e8baa","Type":"ContainerStarted","Data":"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a"} Oct 30 19:53:37.229984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.229485 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" event={"ID":"b8cbf048-c3a9-40b3-b589-5844ba2e8baa","Type":"ContainerStarted","Data":"1a0eed62589aff9c304d80a4cce884a0b752158a3e18e9ccfc34a64f46e3f467"} Oct 30 19:53:37.232108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.231729 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9235-f174a8" Oct 30 19:53:37.232108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.231772 4990 scope.go:122] "RemoveContainer" containerID="eefab0029b50c5f48aebecd64d4fa8681a10af664a3ce366b90afa423781bded" Oct 30 19:53:37.234289 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.234257 4990 generic.go:356] "Generic (PLEG): container finished" podID="2b0618b6-428f-41d5-8c70-161b161ebfe4" containerID="3e399028075b49338d93a4b754a77d462440ea97b027cc638052be07535141a5" exitCode=0 Oct 30 19:53:37.234445 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.234303 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" event={"ID":"2b0618b6-428f-41d5-8c70-161b161ebfe4","Type":"ContainerDied","Data":"3e399028075b49338d93a4b754a77d462440ea97b027cc638052be07535141a5"} Oct 30 19:53:37.264849 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.264820 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:37.266450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.266426 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:37.266571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.266472 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:37.282898 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.282852 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podStartSLOduration=2.282838843 podStartE2EDuration="2.282838843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:37.276826046 +0000 UTC m=+2924.988351415" watchObservedRunningTime="2026-10-30 19:53:37.282838843 +0000 UTC m=+2924.994364278" Oct 30 19:53:37.997784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:37.997369 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.155256 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.155206 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.296101 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:38.307117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.296178 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:38.640395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.640285 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.649767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.649726 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6dfbef8-cfbf-4f66-ba4f-a794d69eb599" path="/var/lib/kubelet/pods/a6dfbef8-cfbf-4f66-ba4f-a794d69eb599/volumes" Oct 30 19:53:38.716294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.715944 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:38.906134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.904631 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\") pod \"2b0618b6-428f-41d5-8c70-161b161ebfe4\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " Oct 30 19:53:38.906134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.904705 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\" (UniqueName: \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\") pod \"2b0618b6-428f-41d5-8c70-161b161ebfe4\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " Oct 30 19:53:38.906134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.904938 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\") pod \"2b0618b6-428f-41d5-8c70-161b161ebfe4\" (UID: \"2b0618b6-428f-41d5-8c70-161b161ebfe4\") " Oct 30 19:53:38.906134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.905997 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca" pod "2b0618b6-428f-41d5-8c70-161b161ebfe4" (UID: "2b0618b6-428f-41d5-8c70-161b161ebfe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:38.924276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.923292 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca" pod "2b0618b6-428f-41d5-8c70-161b161ebfe4" (UID: "2b0618b6-428f-41d5-8c70-161b161ebfe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:38.929117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.928962 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.932296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:38.929847 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57" pod "2b0618b6-428f-41d5-8c70-161b161ebfe4" (UID: "2b0618b6-428f-41d5-8c70-161b161ebfe4"). InnerVolumeSpecName "kube-api-access-zkm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:39.007241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.007166 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:39.007241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.007205 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0618b6-428f-41d5-8c70-161b161ebfe4-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:39.007241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.007217 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkm57\" (UniqueName: \"kubernetes.io/projected/2b0618b6-428f-41d5-8c70-161b161ebfe4-kube-api-access-zkm57\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:39.260399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.260294 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" Oct 30 19:53:39.260399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.260320 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk" event={"ID":"2b0618b6-428f-41d5-8c70-161b161ebfe4","Type":"ContainerDied","Data":"c918f691441af6619742e84713577787d3b9aa012a8656ce9904773e217654c6"} Oct 30 19:53:39.260399 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.260378 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c918f691441af6619742e84713577787d3b9aa012a8656ce9904773e217654c6" Oct 30 19:53:39.283126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.282257 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:39.283126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:39.282326 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:40.117436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.117390 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" podUID="5f94408e-1d81-4171-9385-2b2e933c5ab1" containerName="payload-processing" containerID="cri-o://1b59fdd40a81316b30aa847ed2f6ae864cc3f1d98cb135fe815466eb1f6b0fc8" gracePeriod=25 Oct 30 19:53:40.152255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.152205 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" podUID="324d2b9c-673e-4b00-97df-9a168e4d37d9" containerName="payload-pre-processing" containerID="cri-o://1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413" gracePeriod=25 Oct 30 19:53:40.265508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.265470 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:40.265705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.265568 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:40.305296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.305238 4990 generic.go:356] "Generic (PLEG): container finished" podID="5f94408e-1d81-4171-9385-2b2e933c5ab1" containerID="1b59fdd40a81316b30aa847ed2f6ae864cc3f1d98cb135fe815466eb1f6b0fc8" exitCode=0 Oct 30 19:53:40.305485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.305315 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" event={"ID":"5f94408e-1d81-4171-9385-2b2e933c5ab1","Type":"ContainerDied","Data":"1b59fdd40a81316b30aa847ed2f6ae864cc3f1d98cb135fe815466eb1f6b0fc8"} Oct 30 19:53:40.542145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.542116 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:53:40.647026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.642229 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.717998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.717537 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:53:40.736500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.736468 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\" (UniqueName: \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\") pod \"5f94408e-1d81-4171-9385-2b2e933c5ab1\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " Oct 30 19:53:40.736671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.736631 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\") pod \"5f94408e-1d81-4171-9385-2b2e933c5ab1\" (UID: \"5f94408e-1d81-4171-9385-2b2e933c5ab1\") " Oct 30 19:53:40.737143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.737041 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume" pod "5f94408e-1d81-4171-9385-2b2e933c5ab1" (UID: "5f94408e-1d81-4171-9385-2b2e933c5ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:40.738791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.738765 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9" pod "5f94408e-1d81-4171-9385-2b2e933c5ab1" (UID: "5f94408e-1d81-4171-9385-2b2e933c5ab1"). InnerVolumeSpecName "kube-api-access-652h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:40.838104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838061 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\" (UniqueName: \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\") pod \"324d2b9c-673e-4b00-97df-9a168e4d37d9\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " Oct 30 19:53:40.838318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838298 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\") pod \"324d2b9c-673e-4b00-97df-9a168e4d37d9\" (UID: \"324d2b9c-673e-4b00-97df-9a168e4d37d9\") " Oct 30 19:53:40.838630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838606 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume" pod "324d2b9c-673e-4b00-97df-9a168e4d37d9" (UID: "324d2b9c-673e-4b00-97df-9a168e4d37d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:40.838913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838882 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f94408e-1d81-4171-9385-2b2e933c5ab1-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:40.838913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838899 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324d2b9c-673e-4b00-97df-9a168e4d37d9-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:40.838913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.838908 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-652h9\" (UniqueName: \"kubernetes.io/projected/5f94408e-1d81-4171-9385-2b2e933c5ab1-kube-api-access-652h9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:40.840390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.840365 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn" pod "324d2b9c-673e-4b00-97df-9a168e4d37d9" (UID: "324d2b9c-673e-4b00-97df-9a168e4d37d9"). InnerVolumeSpecName "kube-api-access-mgtwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:40.940179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:40.940128 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtwn\" (UniqueName: \"kubernetes.io/projected/324d2b9c-673e-4b00-97df-9a168e4d37d9-kube-api-access-mgtwn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:41.274117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.273578 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:41.274117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.273660 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:41.318695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.318644 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" event={"ID":"5f94408e-1d81-4171-9385-2b2e933c5ab1","Type":"ContainerDied","Data":"72c2ae3171a2972067853be3f86ab944052b0817f10e78f0013cd1608765534a"} Oct 30 19:53:41.318895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.318661 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477" Oct 30 19:53:41.318895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.318765 4990 scope.go:122] "RemoveContainer" containerID="1b59fdd40a81316b30aa847ed2f6ae864cc3f1d98cb135fe815466eb1f6b0fc8" Oct 30 19:53:41.323643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.323586 4990 generic.go:356] "Generic (PLEG): container finished" podID="324d2b9c-673e-4b00-97df-9a168e4d37d9" containerID="1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413" exitCode=0 Oct 30 19:53:41.323752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.323644 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" Oct 30 19:53:41.323752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.323734 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" event={"ID":"324d2b9c-673e-4b00-97df-9a168e4d37d9","Type":"ContainerDied","Data":"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413"} Oct 30 19:53:41.323902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.323774 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766" event={"ID":"324d2b9c-673e-4b00-97df-9a168e4d37d9","Type":"ContainerDied","Data":"e655a4389c4a740018b96575741d3200bda2eed020f97dea8a01cbcaee40ec2a"} Oct 30 19:53:41.336076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.336054 4990 scope.go:122] "RemoveContainer" containerID="1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413" Oct 30 19:53:41.347718 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.347690 4990 scope.go:122] "RemoveContainer" containerID="1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413" Oct 30 19:53:41.348039 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:53:41.348016 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413\": container with ID starting with 1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413 not found: ID does not exist" containerID="1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413" Oct 30 19:53:41.348133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.348052 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413"} err="failed to get container status \"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413\": rpc error: code = NotFound desc = could not find container \"1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413\": container with ID starting with 1ab9e8deb23247802d6ffb79ec897e3920328e24c001e031885e4aea16c56413 not found: ID does not exist" Oct 30 19:53:41.402915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.402826 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:53:41.407709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.407672 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-7bdb4c69db-h2477"] Oct 30 19:53:41.435810 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.435776 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:53:41.442175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:41.442142 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-54b4fffd95-bf766"] Oct 30 19:53:42.289125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:42.286506 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:42.289125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:42.286579 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:42.412834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:42.412799 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324d2b9c-673e-4b00-97df-9a168e4d37d9" path="/var/lib/kubelet/pods/324d2b9c-673e-4b00-97df-9a168e4d37d9/volumes" Oct 30 19:53:42.413452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:42.413424 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f94408e-1d81-4171-9385-2b2e933c5ab1" path="/var/lib/kubelet/pods/5f94408e-1d81-4171-9385-2b2e933c5ab1/volumes" Oct 30 19:53:42.512117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:42.506049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:43.265617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.265576 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 19:53:43.265807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.265659 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 19:53:43.551823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.551778 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk"] Oct 30 19:53:43.553128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553058 4990 cpu_manager.go:414] "containerMap: removing container" podUID="324d2b9c-673e-4b00-97df-9a168e4d37d9" containerName="payload-pre-processing" Oct 30 19:53:43.553128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553106 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324d2b9c-673e-4b00-97df-9a168e4d37d9" containerName="payload-pre-processing" Oct 30 19:53:43.553333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553211 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5f94408e-1d81-4171-9385-2b2e933c5ab1" containerName="payload-processing" Oct 30 19:53:43.553333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553222 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f94408e-1d81-4171-9385-2b2e933c5ab1" containerName="payload-processing" Oct 30 19:53:43.553333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553252 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2b0618b6-428f-41d5-8c70-161b161ebfe4" containerName="revoke-keys" Oct 30 19:53:43.553333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553261 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0618b6-428f-41d5-8c70-161b161ebfe4" containerName="revoke-keys" Oct 30 19:53:43.553548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553451 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f94408e-1d81-4171-9385-2b2e933c5ab1" containerName="payload-processing" Oct 30 19:53:43.553548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553470 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0618b6-428f-41d5-8c70-161b161ebfe4" containerName="revoke-keys" Oct 30 19:53:43.553548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.553484 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="324d2b9c-673e-4b00-97df-9a168e4d37d9" containerName="payload-pre-processing" Oct 30 19:53:43.557491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.557467 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.580151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.573672 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk"] Oct 30 19:53:43.580151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.575325 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:53:43.580151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.575913 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:43.716999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.716962 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.717292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.717267 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8k7\" (UniqueName: \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.717535 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.717513 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.819509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.819403 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8k7\" (UniqueName: \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.819693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.819580 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.819693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.819625 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.820522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.820485 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.822173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.822118 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.837705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.837671 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8k7\" (UniqueName: \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\") pod \"maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:43.880530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:43.880492 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:44.220022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.219988 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk"] Oct 30 19:53:44.221454 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:53:44.221410 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bfd366_9716_459d_baff_561c88c6e4c5.slice/crio-1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e WatchSource:0}: Error finding container 1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e: Status 404 returned error can't find the container with id 1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e Oct 30 19:53:44.223500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.223465 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.395178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.391497 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:44.395178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.394490 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:44.431658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.431621 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:53:44.432670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.432635 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" event={"ID":"17bfd366-9716-459d-baff-561c88c6e4c5","Type":"ContainerStarted","Data":"1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e"} Oct 30 19:53:44.882642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.882590 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm"] Oct 30 19:53:44.888015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.887962 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:44.904832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:44.904796 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm"] Oct 30 19:53:45.057214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.057174 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.057436 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.057383 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.057483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.057436 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb4w\" (UniqueName: \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.159255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.159146 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.159452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.159307 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.159452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.159348 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb4w\" (UniqueName: \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.159944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.159919 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.161912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.161882 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.179144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.179112 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb4w\" (UniqueName: \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\") pod \"maas-api-revoke-sub-fc3595fd-8drrm\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.220211 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.220166 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:45.443791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.443702 4990 generic.go:356] "Generic (PLEG): container finished" podID="17bfd366-9716-459d-baff-561c88c6e4c5" containerID="43be77d766b9d26dfce8f6034f1bfa402d715540f00b4eeda1a5507fa3238efd" exitCode=0 Oct 30 19:53:45.443955 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.443794 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" event={"ID":"17bfd366-9716-459d-baff-561c88c6e4c5","Type":"ContainerDied","Data":"43be77d766b9d26dfce8f6034f1bfa402d715540f00b4eeda1a5507fa3238efd"} Oct 30 19:53:45.529653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:45.529624 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm"] Oct 30 19:53:45.530400 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:53:45.530376 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfc2b5d_fd55_4e2f_a2f3_059ec1794199.slice/crio-0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c WatchSource:0}: Error finding container 0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c: Status 404 returned error can't find the container with id 0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c Oct 30 19:53:46.166046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.165950 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:46.456561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.456461 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" event={"ID":"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199","Type":"ContainerStarted","Data":"161ea29a5413ecd0af65bf1b6ef99b9e54eecdbd5649e0fe23f39c1385429acd"} Oct 30 19:53:46.456561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.456524 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" event={"ID":"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199","Type":"ContainerStarted","Data":"0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c"} Oct 30 19:53:46.495372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.495311 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" podStartSLOduration=2.495294971 podStartE2EDuration="2.495294971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:46.490870398 +0000 UTC m=+2934.202395764" watchObservedRunningTime="2026-10-30 19:53:46.495294971 +0000 UTC m=+2934.206820337" Oct 30 19:53:46.696152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.696104 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:46.796713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.796621 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\") pod \"17bfd366-9716-459d-baff-561c88c6e4c5\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " Oct 30 19:53:46.796923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.796729 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\" (UniqueName: \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\") pod \"17bfd366-9716-459d-baff-561c88c6e4c5\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " Oct 30 19:53:46.796923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.796760 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\") pod \"17bfd366-9716-459d-baff-561c88c6e4c5\" (UID: \"17bfd366-9716-459d-baff-561c88c6e4c5\") " Oct 30 19:53:46.797264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.797231 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca" pod "17bfd366-9716-459d-baff-561c88c6e4c5" (UID: "17bfd366-9716-459d-baff-561c88c6e4c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:46.797467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.797444 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:46.798997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.798966 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca" pod "17bfd366-9716-459d-baff-561c88c6e4c5" (UID: "17bfd366-9716-459d-baff-561c88c6e4c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:46.798997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.798986 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7" pod "17bfd366-9716-459d-baff-561c88c6e4c5" (UID: "17bfd366-9716-459d-baff-561c88c6e4c5"). InnerVolumeSpecName "kube-api-access-rq8k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:46.899047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.899001 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17bfd366-9716-459d-baff-561c88c6e4c5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:46.899047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:46.899043 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8k7\" (UniqueName: \"kubernetes.io/projected/17bfd366-9716-459d-baff-561c88c6e4c5-kube-api-access-rq8k7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:47.469122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.469066 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" event={"ID":"17bfd366-9716-459d-baff-561c88c6e4c5","Type":"ContainerDied","Data":"1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e"} Oct 30 19:53:47.469561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.469126 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk" Oct 30 19:53:47.469561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.469152 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2dd1294c643ea9881727a4c1eadc460e5040198e544b5fbf22451b2da1069e" Oct 30 19:53:47.471645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.471068 4990 generic.go:356] "Generic (PLEG): container finished" podID="fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" containerID="161ea29a5413ecd0af65bf1b6ef99b9e54eecdbd5649e0fe23f39c1385429acd" exitCode=0 Oct 30 19:53:47.471645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.471197 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" event={"ID":"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199","Type":"ContainerDied","Data":"161ea29a5413ecd0af65bf1b6ef99b9e54eecdbd5649e0fe23f39c1385429acd"} Oct 30 19:53:47.935157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:47.932631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:48.654894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.654861 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:48.730762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.730664 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\") pod \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " Oct 30 19:53:48.730960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.730902 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\") pod \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " Oct 30 19:53:48.731040 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.731011 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\" (UniqueName: \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\") pod \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\" (UID: \"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199\") " Oct 30 19:53:48.731365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.731333 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca" pod "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" (UID: "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:48.731848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.731824 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:48.733584 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.733552 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca" pod "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" (UID: "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:48.733797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.733778 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w" pod "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" (UID: "fbfc2b5d-fd55-4e2f-a2f3-059ec1794199"). InnerVolumeSpecName "kube-api-access-7tb4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:48.833328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.833284 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:48.833328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:48.833329 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tb4w\" (UniqueName: \"kubernetes.io/projected/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199-kube-api-access-7tb4w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:53:49.491077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:49.490885 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:49.508489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:49.508443 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" Oct 30 19:53:49.508665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:49.508446 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm" event={"ID":"fbfc2b5d-fd55-4e2f-a2f3-059ec1794199","Type":"ContainerDied","Data":"0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c"} Oct 30 19:53:49.508665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:49.508598 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5d886afe0383b0382166f4b2d9e5ade3b1500cb4040249cd5dfa1ca972999c" Oct 30 19:53:51.038229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:51.038190 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:52.701780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:52.701737 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:52.850988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:52.850874 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:52.992428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:52.992330 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.710743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:54.710687 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.364208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:56.359765 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.905646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:56.905597 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 19:53:57.255702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:57.255603 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:58.050179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:58.048436 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:59.532588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:59.532546 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:59.689114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:53:59.685866 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.247515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:01.247474 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.042554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:02.042511 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.444740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:02.444700 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.919653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:02.919611 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:04.513416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.513366 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 19:54:04.514533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514505 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" containerName="revoke-keys" Oct 30 19:54:04.514533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514531 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" containerName="revoke-keys" Oct 30 19:54:04.514760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514593 4990 cpu_manager.go:414] "containerMap: removing container" podUID="17bfd366-9716-459d-baff-561c88c6e4c5" containerName="revoke-keys" Oct 30 19:54:04.514760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514599 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bfd366-9716-459d-baff-561c88c6e4c5" containerName="revoke-keys" Oct 30 19:54:04.514760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514736 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" containerName="revoke-keys" Oct 30 19:54:04.514926 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.514765 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bfd366-9716-459d-baff-561c88c6e4c5" containerName="revoke-keys" Oct 30 19:54:04.521318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.521284 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.534776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.534744 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5a0a17-openshift-default-dockercfg-xfwqt\"" Oct 30 19:54:04.554992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.554951 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 19:54:04.604956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.604906 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605045 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605220 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605337 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605394 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605435 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605553 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605710 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxtl\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.605885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.605761 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.650733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.650681 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:04.707639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.707591 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.707834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.707722 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.707834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.707768 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.707952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.707915 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708046 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708059 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708128 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708278 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxtl\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708323 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708406 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.708664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708561 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.709007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.708981 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.709187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.709108 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.710773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.710745 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.710942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.710920 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.725859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.725814 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.726022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.725885 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxtl\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:04.838169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:04.838135 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:05.100149 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:05.100032 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:54:05.177630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.177596 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 19:54:05.178465 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:05.178436 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e283a85_0123_4942_9b49_84b639f8b5ee.slice/crio-c35ef7ef50c01718e89b7dd8d0811a3de413be397d27d4afa5d4074865fb8ebf WatchSource:0}: Error finding container c35ef7ef50c01718e89b7dd8d0811a3de413be397d27d4afa5d4074865fb8ebf: Status 404 returned error can't find the container with id c35ef7ef50c01718e89b7dd8d0811a3de413be397d27d4afa5d4074865fb8ebf Oct 30 19:54:05.182159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.182117 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:05.182282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.182207 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:05.182282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.182238 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:05.670548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.670507 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" event={"ID":"5e283a85-0123-4942-9b49-84b639f8b5ee","Type":"ContainerStarted","Data":"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f"} Oct 30 19:54:05.671104 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.670579 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" event={"ID":"5e283a85-0123-4942-9b49-84b639f8b5ee","Type":"ContainerStarted","Data":"c35ef7ef50c01718e89b7dd8d0811a3de413be397d27d4afa5d4074865fb8ebf"} Oct 30 19:54:05.729812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.729757 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podStartSLOduration=1.7297400920000001 podStartE2EDuration="1.729740092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:05.718344638 +0000 UTC m=+2953.429870004" watchObservedRunningTime="2026-10-30 19:54:05.729740092 +0000 UTC m=+2953.441265458" Oct 30 19:54:05.838659 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:05.838628 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:06.210322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:06.210272 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.839960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:06.839928 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:06.840449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:06.839984 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:07.838754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:07.838707 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:07.838998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:07.838802 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:07.999233 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:07.998793 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:08.839108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:08.839054 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:08.839360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:08.839135 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:09.277481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:09.273950 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:54:09.326112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:09.326008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.495658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:09.495620 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.839121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:09.838547 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:09.839121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:09.838641 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:10.006514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:10.006473 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:10.620285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:10.620235 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:10.791117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:10.790693 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:10.838986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:10.838951 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:10.839217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:10.839022 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:11.353775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:11.353731 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:11.839202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:11.839160 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 19:54:11.839616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:11.839231 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 19:54:12.845949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:12.845916 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:12.846507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:12.846488 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:12.847416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:12.847393 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 19:54:13.482135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:13.482062 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.840759 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:13.840722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.993806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:13.993767 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.277834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:15.277633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.527714 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:15.527463 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.832125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:15.832056 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 19:54:15.895156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:15.890446 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.083753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:15.895316 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 19:54:16.136122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.111460 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br6w\" (UniqueName: \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.158388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.158346 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.158916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.136414 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ed2ede-dockercfg-rwfdf\"" Oct 30 19:54:16.159058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.113806 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ed2ede\"" Oct 30 19:54:16.262697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.262655 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br6w\" (UniqueName: \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.263473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.263446 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.265944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.265596 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.304841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.304800 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br6w\" (UniqueName: \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\") pod \"payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.457308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.457213 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:16.670675 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.670635 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.769419 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:16.769385 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4247cd_4e68_4443_8fe8_3f10e59ae5ff.slice/crio-89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538 WatchSource:0}: Error finding container 89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538: Status 404 returned error can't find the container with id 89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538 Oct 30 19:54:16.770910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.770880 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 19:54:16.797956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.797925 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:54:16.804722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.804703 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:16.820758 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.820723 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf314f\"" Oct 30 19:54:16.820967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.820916 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf314f-dockercfg-mf9d7\"" Oct 30 19:54:16.823225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.823201 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:54:16.829988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.829956 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.875943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.875908 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwst\" (UniqueName: \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:16.876450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:16.876420 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.036520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.015152 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwst\" (UniqueName: \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.036520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.016370 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.161151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.041148 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:54:17.161151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.094632 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwst\" (UniqueName: \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.161151 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.150504 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.165236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.165076 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.167941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.167915 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:17.176528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.175368 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" event={"ID":"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff","Type":"ContainerStarted","Data":"89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538"} Oct 30 19:54:17.176528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.175603 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.179743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.179719 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:54:17.270143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.269908 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcrf\" (UniqueName: \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.271118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.270953 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.372913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.372862 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcrf\" (UniqueName: \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.373774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.373744 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.375077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.375050 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.398401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.398328 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcrf\" (UniqueName: \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\") pod \"payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.514775 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.514734 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:17.521910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.521872 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:54:17.523199 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:17.523167 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f5b116_f17d_46f4_8d0d_80ddd51e6690.slice/crio-350102254a2e4fb53e2cc7ee558c8d4e05aebbb83c31095898d36f3da8a19ceb WatchSource:0}: Error finding container 350102254a2e4fb53e2cc7ee558c8d4e05aebbb83c31095898d36f3da8a19ceb: Status 404 returned error can't find the container with id 350102254a2e4fb53e2cc7ee558c8d4e05aebbb83c31095898d36f3da8a19ceb Oct 30 19:54:17.852394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:17.852365 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:54:17.856290 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:17.856251 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b29685_2d71_46fe_a51d_b0f12d864fa4.slice/crio-32c834593d5f726178446979face21ee248f1d0321cdd285eaa74766cd4765f3 WatchSource:0}: Error finding container 32c834593d5f726178446979face21ee248f1d0321cdd285eaa74766cd4765f3: Status 404 returned error can't find the container with id 32c834593d5f726178446979face21ee248f1d0321cdd285eaa74766cd4765f3 Oct 30 19:54:18.130474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.130438 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" event={"ID":"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff","Type":"ContainerStarted","Data":"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391"} Oct 30 19:54:18.130691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.130557 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:18.132318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.132292 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" event={"ID":"d3b29685-2d71-46fe-a51d-b0f12d864fa4","Type":"ContainerStarted","Data":"189ae794e7232fe6dabe1cd5e70a9878f17244520e9c68f223ac97daf1ff86ca"} Oct 30 19:54:18.132456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.132338 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" event={"ID":"d3b29685-2d71-46fe-a51d-b0f12d864fa4","Type":"ContainerStarted","Data":"32c834593d5f726178446979face21ee248f1d0321cdd285eaa74766cd4765f3"} Oct 30 19:54:18.132456 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.132415 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:18.134232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.134204 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" event={"ID":"f8f5b116-f17d-46f4-8d0d-80ddd51e6690","Type":"ContainerStarted","Data":"75d978e34d145e6a2e7b922e5ea4e526d1e0603b1eeef334519f0d70c9ecdf52"} Oct 30 19:54:18.134357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.134243 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" event={"ID":"f8f5b116-f17d-46f4-8d0d-80ddd51e6690","Type":"ContainerStarted","Data":"350102254a2e4fb53e2cc7ee558c8d4e05aebbb83c31095898d36f3da8a19ceb"} Oct 30 19:54:18.134357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.134273 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:18.170677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.170611 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" podStartSLOduration=3.170591365 podStartE2EDuration="3.170591365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:18.164507924 +0000 UTC m=+2965.876033290" watchObservedRunningTime="2026-10-30 19:54:18.170591365 +0000 UTC m=+2965.882116732" Oct 30 19:54:18.200215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.200080 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" podStartSLOduration=2.200065454 podStartE2EDuration="2.200065454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:18.199039066 +0000 UTC m=+2965.910564432" watchObservedRunningTime="2026-10-30 19:54:18.200065454 +0000 UTC m=+2965.911590819" Oct 30 19:54:18.235556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.235504 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" podStartSLOduration=2.235489672 podStartE2EDuration="2.235489672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:18.232076492 +0000 UTC m=+2965.943601861" watchObservedRunningTime="2026-10-30 19:54:18.235489672 +0000 UTC m=+2965.947015037" Oct 30 19:54:18.945111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:18.943989 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:20.838085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:20.838047 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.261057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.261016 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.316809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.316736 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:54:22.317057 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.317029 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" podUID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" containerName="payload-processing" containerID="cri-o://f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573" gracePeriod=30 Oct 30 19:54:22.330230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.330202 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:54:22.330431 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.330406 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" podUID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" containerName="payload-pre-processing" containerID="cri-o://f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c" gracePeriod=30 Oct 30 19:54:22.432514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.432476 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.566636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.566596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.772183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.761746 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:54:22.783135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.780354 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.847130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.839739 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:54:22.884795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.884743 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\") pod \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " Oct 30 19:54:22.884996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.884884 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\" (UniqueName: \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\") pod \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " Oct 30 19:54:22.885146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.885067 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\" (UniqueName: \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\") pod \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\" (UID: \"7d2dd0ae-31f2-46b1-a90d-2e82e024178b\") " Oct 30 19:54:22.885213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.885180 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\") pod \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\" (UID: \"ca5cdadf-9fda-4c36-8843-dceacba7a5cd\") " Oct 30 19:54:22.887308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.887275 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume" pod "7d2dd0ae-31f2-46b1-a90d-2e82e024178b" (UID: "7d2dd0ae-31f2-46b1-a90d-2e82e024178b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:22.887503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.887479 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume" pod "ca5cdadf-9fda-4c36-8843-dceacba7a5cd" (UID: "ca5cdadf-9fda-4c36-8843-dceacba7a5cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:22.908889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.908806 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc" pod "7d2dd0ae-31f2-46b1-a90d-2e82e024178b" (UID: "7d2dd0ae-31f2-46b1-a90d-2e82e024178b"). InnerVolumeSpecName "kube-api-access-trfnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:22.908889 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.908841 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj" pod "ca5cdadf-9fda-4c36-8843-dceacba7a5cd" (UID: "ca5cdadf-9fda-4c36-8843-dceacba7a5cd"). InnerVolumeSpecName "kube-api-access-gqfkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:22.973142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.973064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.977166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.977137 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk"] Oct 30 19:54:22.984348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.984325 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-852709-gpmjk"] Oct 30 19:54:22.988053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.988027 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfkj\" (UniqueName: \"kubernetes.io/projected/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-kube-api-access-gqfkj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:22.988193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.988054 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfnc\" (UniqueName: \"kubernetes.io/projected/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-kube-api-access-trfnc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:22.988193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.988070 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5cdadf-9fda-4c36-8843-dceacba7a5cd-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:22.988193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:22.988084 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2dd0ae-31f2-46b1-a90d-2e82e024178b-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:23.124109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.121473 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.252177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.252142 4990 generic.go:356] "Generic (PLEG): container finished" podID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" containerID="f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573" exitCode=0 Oct 30 19:54:23.252350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.252189 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" Oct 30 19:54:23.252350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.252224 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" event={"ID":"7d2dd0ae-31f2-46b1-a90d-2e82e024178b","Type":"ContainerDied","Data":"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573"} Oct 30 19:54:23.252350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.252284 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs" event={"ID":"7d2dd0ae-31f2-46b1-a90d-2e82e024178b","Type":"ContainerDied","Data":"65a71b60f7d25335d0829258c930f5d2277f36c81fed88a9a69b0b2b077f0266"} Oct 30 19:54:23.252350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.252313 4990 scope.go:122] "RemoveContainer" containerID="f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573" Oct 30 19:54:23.254066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.254034 4990 generic.go:356] "Generic (PLEG): container finished" podID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" containerID="f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c" exitCode=0 Oct 30 19:54:23.254231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.254105 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" event={"ID":"ca5cdadf-9fda-4c36-8843-dceacba7a5cd","Type":"ContainerDied","Data":"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c"} Oct 30 19:54:23.254231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.254085 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" Oct 30 19:54:23.254231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.254158 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755" event={"ID":"ca5cdadf-9fda-4c36-8843-dceacba7a5cd","Type":"ContainerDied","Data":"d7e331469d0624177730cf79e549c7319b1973ff2d466bb02b2e3732133dc125"} Oct 30 19:54:23.266207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.266181 4990 scope.go:122] "RemoveContainer" containerID="f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573" Oct 30 19:54:23.266536 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:23.266511 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573\": container with ID starting with f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573 not found: ID does not exist" containerID="f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573" Oct 30 19:54:23.266594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.266548 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573"} err="failed to get container status \"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573\": rpc error: code = NotFound desc = could not find container \"f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573\": container with ID starting with f7870e711e6dfbe11e14c4393eeb16074498d5ce6bfa2d33701b330c273fc573 not found: ID does not exist" Oct 30 19:54:23.266594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.266572 4990 scope.go:122] "RemoveContainer" containerID="f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c" Oct 30 19:54:23.278241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.278218 4990 scope.go:122] "RemoveContainer" containerID="f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c" Oct 30 19:54:23.278553 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:23.278523 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c\": container with ID starting with f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c not found: ID does not exist" containerID="f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c" Oct 30 19:54:23.278633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.278561 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c"} err="failed to get container status \"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c\": rpc error: code = NotFound desc = could not find container \"f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c\": container with ID starting with f2ff001483396766afa13f5f8d3bd262f6685c63afd4feadec2aae8323446b3c not found: ID does not exist" Oct 30 19:54:23.312360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.312322 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:54:23.318350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.318319 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-852709-d58bf4df8-vp8zs"] Oct 30 19:54:23.346224 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.346170 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:54:23.351263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.351233 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-852709-545f78f65f-wb755"] Oct 30 19:54:23.515968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:23.515889 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.334078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.334038 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" podUID="78e67769-a212-42da-8082-1f1c56129c31" containerName="main" containerID="cri-o://f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148" gracePeriod=45 Oct 30 19:54:24.404987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.404951 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bfd366-9716-459d-baff-561c88c6e4c5" path="/var/lib/kubelet/pods/17bfd366-9716-459d-baff-561c88c6e4c5/volumes" Oct 30 19:54:24.405504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.405486 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" path="/var/lib/kubelet/pods/7d2dd0ae-31f2-46b1-a90d-2e82e024178b/volumes" Oct 30 19:54:24.405881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.405868 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" path="/var/lib/kubelet/pods/ca5cdadf-9fda-4c36-8843-dceacba7a5cd/volumes" Oct 30 19:54:24.645805 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.645774 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:54:24.715606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715559 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.715801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715639 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\" (UniqueName: \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.715801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715697 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\" (UniqueName: \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.715894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715820 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.715894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715885 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.715995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.715929 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\") pod \"78e67769-a212-42da-8082-1f1c56129c31\" (UID: \"78e67769-a212-42da-8082-1f1c56129c31\") " Oct 30 19:54:24.716303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716273 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:24.716437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716385 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:24.716515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716463 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:24.716894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716859 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.716894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716891 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.717059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.716905 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.718980 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.718947 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:24.719076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.718971 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "kube-api-access-x5v4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:24.719391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.719368 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs" pod "78e67769-a212-42da-8082-1f1c56129c31" (UID: "78e67769-a212-42da-8082-1f1c56129c31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:24.800457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.799789 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.818786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.818579 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e67769-a212-42da-8082-1f1c56129c31-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.818786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.818612 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e67769-a212-42da-8082-1f1c56129c31-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:24.818786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:24.818627 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5v4l\" (UniqueName: \"kubernetes.io/projected/78e67769-a212-42da-8082-1f1c56129c31-kube-api-access-x5v4l\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:25.277571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.277528 4990 generic.go:356] "Generic (PLEG): container finished" podID="78e67769-a212-42da-8082-1f1c56129c31" containerID="f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148" exitCode=0 Oct 30 19:54:25.277801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.277578 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" Oct 30 19:54:25.277801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.277579 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" event={"ID":"78e67769-a212-42da-8082-1f1c56129c31","Type":"ContainerDied","Data":"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148"} Oct 30 19:54:25.277801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.277689 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9" event={"ID":"78e67769-a212-42da-8082-1f1c56129c31","Type":"ContainerDied","Data":"f2838faf37b4e26c99351615e203908c59a8b4c1286acd09bac79fc0c355e61d"} Oct 30 19:54:25.277801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.277719 4990 scope.go:122] "RemoveContainer" containerID="f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148" Oct 30 19:54:25.292180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.292158 4990 scope.go:122] "RemoveContainer" containerID="f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148" Oct 30 19:54:25.292502 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:25.292480 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148\": container with ID starting with f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148 not found: ID does not exist" containerID="f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148" Oct 30 19:54:25.292576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.292509 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148"} err="failed to get container status \"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148\": rpc error: code = NotFound desc = could not find container \"f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148\": container with ID starting with f42634c5f6b7de0bf5fa26d0c4127cce5ef85703337f54d3eac0f5fad8951148 not found: ID does not exist" Oct 30 19:54:25.325978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.325947 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:54:25.335565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:25.335535 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-287ea2/auth-test-model-287ea2-kserve-899f4d757-wcwb9"] Oct 30 19:54:26.404143 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:26.404110 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e67769-a212-42da-8082-1f1c56129c31" path="/var/lib/kubelet/pods/78e67769-a212-42da-8082-1f1c56129c31/volumes" Oct 30 19:54:26.531790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:26.531083 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.155029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:28.154967 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.146750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.146720 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 19:54:29.147239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.147216 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:54:29.147360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.147336 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:54:29.795532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.795492 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9247-cdc9f2"] Oct 30 19:54:29.796446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796422 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" containerName="payload-processing" Oct 30 19:54:29.796446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796447 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" containerName="payload-processing" Oct 30 19:54:29.796633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796531 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" containerName="payload-pre-processing" Oct 30 19:54:29.796633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796540 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" containerName="payload-pre-processing" Oct 30 19:54:29.796633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796573 4990 cpu_manager.go:414] "containerMap: removing container" podUID="78e67769-a212-42da-8082-1f1c56129c31" containerName="main" Oct 30 19:54:29.796633 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796580 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e67769-a212-42da-8082-1f1c56129c31" containerName="main" Oct 30 19:54:29.796872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796781 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5cdadf-9fda-4c36-8843-dceacba7a5cd" containerName="payload-pre-processing" Oct 30 19:54:29.796872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796807 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e67769-a212-42da-8082-1f1c56129c31" containerName="main" Oct 30 19:54:29.796872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.796822 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2dd0ae-31f2-46b1-a90d-2e82e024178b" containerName="payload-processing" Oct 30 19:54:29.813875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.813844 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:29.821203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.821171 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9247-cdc9f2"] Oct 30 19:54:29.830634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.830598 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-66q2n\"" Oct 30 19:54:29.845084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.845034 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.908846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:29.908803 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq25\" (UniqueName: \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\") pod \"test-curl-9247-cdc9f2\" (UID: \"db985c2b-a0e6-4aeb-b9f6-4085ace14060\") " pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:30.004741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:30.004698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:30.010314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:30.010287 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq25\" (UniqueName: \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\") pod \"test-curl-9247-cdc9f2\" (UID: \"db985c2b-a0e6-4aeb-b9f6-4085ace14060\") " pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:30.027396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:30.027365 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq25\" (UniqueName: \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\") pod \"test-curl-9247-cdc9f2\" (UID: \"db985c2b-a0e6-4aeb-b9f6-4085ace14060\") " pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:30.149410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:30.149366 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:30.463885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:30.463848 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9247-cdc9f2"] Oct 30 19:54:30.464993 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:30.464950 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb985c2b_a0e6_4aeb_b9f6_4085ace14060.slice/crio-b867c3c5978cb2ed2fa0dc4b9447a623c95949b9192914dc3a0558dcbe200166 WatchSource:0}: Error finding container b867c3c5978cb2ed2fa0dc4b9447a623c95949b9192914dc3a0558dcbe200166: Status 404 returned error can't find the container with id b867c3c5978cb2ed2fa0dc4b9447a623c95949b9192914dc3a0558dcbe200166 Oct 30 19:54:31.356401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:31.356361 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9247-cdc9f2" event={"ID":"db985c2b-a0e6-4aeb-b9f6-4085ace14060","Type":"ContainerStarted","Data":"cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2"} Oct 30 19:54:31.356834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:31.356412 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9247-cdc9f2" event={"ID":"db985c2b-a0e6-4aeb-b9f6-4085ace14060","Type":"ContainerStarted","Data":"b867c3c5978cb2ed2fa0dc4b9447a623c95949b9192914dc3a0558dcbe200166"} Oct 30 19:54:31.394318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:31.394253 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9247-cdc9f2" podStartSLOduration=2.394234307 podStartE2EDuration="2.394234307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:31.39072105 +0000 UTC m=+2979.102246416" watchObservedRunningTime="2026-10-30 19:54:31.394234307 +0000 UTC m=+2979.105759672" Oct 30 19:54:31.830067 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:31.830033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:32.257362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.257265 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9247-cdc9f2"] Oct 30 19:54:32.262374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.262344 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9247-cdc9f2"] Oct 30 19:54:32.413923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.413889 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9247-78f35d"] Oct 30 19:54:32.414823 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.414798 4990 cpu_manager.go:414] "containerMap: removing container" podUID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" containerName="test-curl-9247-cdc9f2" Oct 30 19:54:32.414941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.414824 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" containerName="test-curl-9247-cdc9f2" Oct 30 19:54:32.415125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.415074 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" containerName="test-curl-9247-cdc9f2" Oct 30 19:54:32.418911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.418890 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:32.436279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.436253 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9247-78f35d"] Oct 30 19:54:32.549737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.549703 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplzv\" (UniqueName: \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\") pod \"test-curl-9247-78f35d\" (UID: \"44d8e06f-5f03-44cf-9335-5d61828ffc87\") " pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:32.651551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.651516 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplzv\" (UniqueName: \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\") pod \"test-curl-9247-78f35d\" (UID: \"44d8e06f-5f03-44cf-9335-5d61828ffc87\") " pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:32.669039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.669009 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplzv\" (UniqueName: \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\") pod \"test-curl-9247-78f35d\" (UID: \"44d8e06f-5f03-44cf-9335-5d61828ffc87\") " pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:32.732546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.732516 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:32.970982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:32.970945 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.048029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.048000 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9247-78f35d"] Oct 30 19:54:33.050114 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:33.050072 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d8e06f_5f03_44cf_9335_5d61828ffc87.slice/crio-a9ffe25d423c83ece9bbfd97730acc116d7161117c693d83750217ed1ebcb06a WatchSource:0}: Error finding container a9ffe25d423c83ece9bbfd97730acc116d7161117c693d83750217ed1ebcb06a: Status 404 returned error can't find the container with id a9ffe25d423c83ece9bbfd97730acc116d7161117c693d83750217ed1ebcb06a Oct 30 19:54:33.593696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.590518 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.658957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.658909 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9247-78f35d" event={"ID":"44d8e06f-5f03-44cf-9335-5d61828ffc87","Type":"ContainerStarted","Data":"0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09"} Oct 30 19:54:33.659261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.658912 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9247-cdc9f2" podUID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" containerName="test-curl-9247-cdc9f2" containerID="cri-o://cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2" gracePeriod=2 Oct 30 19:54:33.659518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.659233 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9247-78f35d" event={"ID":"44d8e06f-5f03-44cf-9335-5d61828ffc87","Type":"ContainerStarted","Data":"a9ffe25d423c83ece9bbfd97730acc116d7161117c693d83750217ed1ebcb06a"} Oct 30 19:54:33.697612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.697546 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9247-78f35d" podStartSLOduration=1.697530371 podStartE2EDuration="1.697530371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:33.691502001 +0000 UTC m=+2981.403027370" watchObservedRunningTime="2026-10-30 19:54:33.697530371 +0000 UTC m=+2981.409055737" Oct 30 19:54:33.750138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.750085 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.907394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:33.907288 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:34.528852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:34.528802 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9247-78f35d"] Oct 30 19:54:34.534293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:34.534264 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9247-78f35d"] Oct 30 19:54:35.100785 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:35.100747 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:54:35.124043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.124011 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j"] Oct 30 19:54:35.124909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.124886 4990 cpu_manager.go:414] "containerMap: removing container" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" containerName="test-curl-9247-78f35d" Oct 30 19:54:35.124909 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.124908 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" containerName="test-curl-9247-78f35d" Oct 30 19:54:35.125119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.125083 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" containerName="test-curl-9247-78f35d" Oct 30 19:54:35.129097 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.129066 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.147449 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.147420 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j"] Oct 30 19:54:35.150604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.150584 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:35.150777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.150763 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:54:35.217117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.217054 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.217352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.217291 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.217587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.217562 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phr79\" (UniqueName: \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.319279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.319240 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.319502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.319408 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phr79\" (UniqueName: \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.319502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.319477 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.320238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.320214 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.321879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.321852 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.337232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.337206 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phr79\" (UniqueName: \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\") pod \"maas-api-revoke-sub-d5fabdda-67q9j\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.374417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.374310 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:35.444925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.444889 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:35.703255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.703214 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9247-78f35d" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" containerName="test-curl-9247-78f35d" containerID="cri-o://0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09" gracePeriod=2 Oct 30 19:54:35.714635 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:35.714596 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9247-78f35d\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:35.758429 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.758394 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j"] Oct 30 19:54:35.759454 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:35.759429 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7855042_e67b_441c_b7b9_49e47aea7015.slice/crio-ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84 WatchSource:0}: Error finding container ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84: Status 404 returned error can't find the container with id ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84 Oct 30 19:54:35.977350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:35.977323 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:35.989050 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:35.989008 4990 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9247-78f35d\" is forbidden: User \"system:node:ip-10-0-135-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-68.ec2.internal' and this object" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:36.034776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.034733 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\" (UniqueName: \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\") pod \"db985c2b-a0e6-4aeb-b9f6-4085ace14060\" (UID: \"db985c2b-a0e6-4aeb-b9f6-4085ace14060\") " Oct 30 19:54:36.037346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.037317 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25" pod "db985c2b-a0e6-4aeb-b9f6-4085ace14060" (UID: "db985c2b-a0e6-4aeb-b9f6-4085ace14060"). InnerVolumeSpecName "kube-api-access-jpq25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:36.137886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.137845 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpq25\" (UniqueName: \"kubernetes.io/projected/db985c2b-a0e6-4aeb-b9f6-4085ace14060-kube-api-access-jpq25\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:36.406526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.406487 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" path="/var/lib/kubelet/pods/db985c2b-a0e6-4aeb-b9f6-4085ace14060/volumes" Oct 30 19:54:36.713749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.713715 4990 generic.go:356] "Generic (PLEG): container finished" podID="db985c2b-a0e6-4aeb-b9f6-4085ace14060" containerID="cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2" exitCode=137 Oct 30 19:54:36.713970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.713761 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-cdc9f2" Oct 30 19:54:36.713970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.713819 4990 scope.go:122] "RemoveContainer" containerID="cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2" Oct 30 19:54:36.716875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.716836 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" event={"ID":"b7855042-e67b-441c-b7b9-49e47aea7015","Type":"ContainerStarted","Data":"b45215b2e07c7af77c1a3fcd6fe1374912570c79a89bc99971561e230f39b14e"} Oct 30 19:54:36.717105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.717055 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" event={"ID":"b7855042-e67b-441c-b7b9-49e47aea7015","Type":"ContainerStarted","Data":"ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84"} Oct 30 19:54:36.725746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.725713 4990 scope.go:122] "RemoveContainer" containerID="cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2" Oct 30 19:54:36.726181 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:36.726086 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2\": container with ID starting with cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2 not found: ID does not exist" containerID="cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2" Oct 30 19:54:36.726181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.726155 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2"} err="failed to get container status \"cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2\": rpc error: code = NotFound desc = could not find container \"cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2\": container with ID starting with cf6c3751f4ef75e4407f267ef76ab7a46c0cef391f8d87447d9a0a876f7a98f2 not found: ID does not exist" Oct 30 19:54:36.755699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:36.755646 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" podStartSLOduration=1.755629362 podStartE2EDuration="1.755629362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:36.750912688 +0000 UTC m=+2984.462438055" watchObservedRunningTime="2026-10-30 19:54:36.755629362 +0000 UTC m=+2984.467154722" Oct 30 19:54:37.011542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:37.011451 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.226112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:37.220025 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.756696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:37.756628 4990 generic.go:356] "Generic (PLEG): container finished" podID="b7855042-e67b-441c-b7b9-49e47aea7015" containerID="b45215b2e07c7af77c1a3fcd6fe1374912570c79a89bc99971561e230f39b14e" exitCode=0 Oct 30 19:54:37.757681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:37.757640 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" event={"ID":"b7855042-e67b-441c-b7b9-49e47aea7015","Type":"ContainerDied","Data":"b45215b2e07c7af77c1a3fcd6fe1374912570c79a89bc99971561e230f39b14e"} Oct 30 19:54:37.955242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:37.955216 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:38.055768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.055733 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\" (UniqueName: \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\") pod \"44d8e06f-5f03-44cf-9335-5d61828ffc87\" (UID: \"44d8e06f-5f03-44cf-9335-5d61828ffc87\") " Oct 30 19:54:38.058269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.058240 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv" pod "44d8e06f-5f03-44cf-9335-5d61828ffc87" (UID: "44d8e06f-5f03-44cf-9335-5d61828ffc87"). InnerVolumeSpecName "kube-api-access-gplzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:38.157464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.157427 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplzv\" (UniqueName: \"kubernetes.io/projected/44d8e06f-5f03-44cf-9335-5d61828ffc87-kube-api-access-gplzv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:38.177673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.177639 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 19:54:38.182626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.182605 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.205543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.205505 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 19:54:38.213107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.208590 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-ec2fe7-openshift-default-dockercfg-c97rm\"" Oct 30 19:54:38.258338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258305 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.258338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258341 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258518 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258624 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258653 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvllg\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258790 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258839 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.259275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.258958 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.360882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.360779 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.360882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.360865 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvllg\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.360965 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361015 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361125 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361192 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361215 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361333 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361359 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361485 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361589 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361707 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361795 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.361936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.361912 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.363751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.363645 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.363901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.363711 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.381340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.381308 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.381593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.381564 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvllg\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.406469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.406425 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d8e06f-5f03-44cf-9335-5d61828ffc87" path="/var/lib/kubelet/pods/44d8e06f-5f03-44cf-9335-5d61828ffc87/volumes" Oct 30 19:54:38.497397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.497361 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:38.782602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.782507 4990 generic.go:356] "Generic (PLEG): container finished" podID="44d8e06f-5f03-44cf-9335-5d61828ffc87" containerID="0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09" exitCode=137 Oct 30 19:54:38.782602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.782546 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9247-78f35d" Oct 30 19:54:38.782853 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.782829 4990 scope.go:122] "RemoveContainer" containerID="0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09" Oct 30 19:54:38.783012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.782860 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.795529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.795504 4990 scope.go:122] "RemoveContainer" containerID="0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09" Oct 30 19:54:38.795904 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:54:38.795870 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09\": container with ID starting with 0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09 not found: ID does not exist" containerID="0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09" Oct 30 19:54:38.795993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.795913 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09"} err="failed to get container status \"0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09\": rpc error: code = NotFound desc = could not find container \"0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09\": container with ID starting with 0159ddbb7bd77d15441ca003fa545f54a2bcd3b9b99b964a8a9ea0cdabd5dd09 not found: ID does not exist" Oct 30 19:54:38.842641 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:38.842601 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75374cca_d96f_4bf1_93dd_ce57969e9f75.slice/crio-dbc88d5064cb3f6f7c837b1004fbffad7450d644ea031919295df323417e48b5 WatchSource:0}: Error finding container dbc88d5064cb3f6f7c837b1004fbffad7450d644ea031919295df323417e48b5: Status 404 returned error can't find the container with id dbc88d5064cb3f6f7c837b1004fbffad7450d644ea031919295df323417e48b5 Oct 30 19:54:38.844881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.844837 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 19:54:38.845856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.845821 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:38.845974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.845884 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:38.845974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.845919 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:54:38.923624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.923586 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:54:38.942873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.939381 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:38.942873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.941483 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:54:38.955264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.952935 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2afc11\"" Oct 30 19:54:38.955884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.955849 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.959818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:38.959792 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2afc11-dockercfg-h8rkx\"" Oct 30 19:54:39.126744 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.123536 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:54:39.128237 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.128211 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:39.180244 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.180207 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:54:39.180460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.180337 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.196116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.193324 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcrr\" (UniqueName: \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.196116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.193457 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.295002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.294642 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\" (UniqueName: \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\") pod \"b7855042-e67b-441c-b7b9-49e47aea7015\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " Oct 30 19:54:39.295503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.295083 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\") pod \"b7855042-e67b-441c-b7b9-49e47aea7015\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " Oct 30 19:54:39.295503 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.295333 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\") pod \"b7855042-e67b-441c-b7b9-49e47aea7015\" (UID: \"b7855042-e67b-441c-b7b9-49e47aea7015\") " Oct 30 19:54:39.295877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.295848 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcrr\" (UniqueName: \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.295979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.295921 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.295979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.295968 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.296258 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.296234 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfqx\" (UniqueName: \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.298045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.298018 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca" pod "b7855042-e67b-441c-b7b9-49e47aea7015" (UID: "b7855042-e67b-441c-b7b9-49e47aea7015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:39.307170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.298060 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.308358 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.308310 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79" pod "b7855042-e67b-441c-b7b9-49e47aea7015" (UID: "b7855042-e67b-441c-b7b9-49e47aea7015"). InnerVolumeSpecName "kube-api-access-phr79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:39.308817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.308776 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca" pod "b7855042-e67b-441c-b7b9-49e47aea7015" (UID: "b7855042-e67b-441c-b7b9-49e47aea7015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:39.313437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.313410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcrr\" (UniqueName: \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.398290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.398199 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfqx\" (UniqueName: \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.398534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.398509 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.398691 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.398676 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phr79\" (UniqueName: \"kubernetes.io/projected/b7855042-e67b-441c-b7b9-49e47aea7015-kube-api-access-phr79\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.398766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.398704 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.398766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.398720 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7855042-e67b-441c-b7b9-49e47aea7015-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.399282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.399257 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.415765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.415729 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfqx\" (UniqueName: \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\") pod \"payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.425487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.425447 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:39.459577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.459488 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.538773 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.537882 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:39.795729 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:39.795685 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6135aae8_d2e3_41b6_860b_50a7c13fbd7d.slice/crio-4ee4c3cc505a9f36845350442d677fe86d2407f62cc12f9278d29977fddb9152 WatchSource:0}: Error finding container 4ee4c3cc505a9f36845350442d677fe86d2407f62cc12f9278d29977fddb9152: Status 404 returned error can't find the container with id 4ee4c3cc505a9f36845350442d677fe86d2407f62cc12f9278d29977fddb9152 Oct 30 19:54:39.822740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.822695 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" event={"ID":"75374cca-d96f-4bf1-93dd-ce57969e9f75","Type":"ContainerStarted","Data":"c0ec7dca8f283d4a8f5bedaaa0a1d9cfaa63c567aa960b9faff565592288075c"} Oct 30 19:54:39.822891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.822796 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" event={"ID":"75374cca-d96f-4bf1-93dd-ce57969e9f75","Type":"ContainerStarted","Data":"dbc88d5064cb3f6f7c837b1004fbffad7450d644ea031919295df323417e48b5"} Oct 30 19:54:39.823660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.823631 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:54:39.826516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.826485 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" event={"ID":"b7855042-e67b-441c-b7b9-49e47aea7015","Type":"ContainerDied","Data":"ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84"} Oct 30 19:54:39.826650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.826540 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff05e20bc47a09c989a98fbc4883eda1850d1a8f6ffbe492f0c2b18b78ac2a84" Oct 30 19:54:39.826650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.826617 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j" Oct 30 19:54:39.867129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.867045 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podStartSLOduration=1.8670202420000002 podStartE2EDuration="1.867020242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:39.860074047 +0000 UTC m=+2987.571599414" watchObservedRunningTime="2026-10-30 19:54:39.867020242 +0000 UTC m=+2987.578545623" Oct 30 19:54:39.933603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:39.933553 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:54:39.955676 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:39.955639 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33c656_b26e_429c_bbc2_2f76974ffd11.slice/crio-ffb6c44986446845a39cdc805f23e19e9ea0a1583299087c64dd13042af978c6 WatchSource:0}: Error finding container ffb6c44986446845a39cdc805f23e19e9ea0a1583299087c64dd13042af978c6: Status 404 returned error can't find the container with id ffb6c44986446845a39cdc805f23e19e9ea0a1583299087c64dd13042af978c6 Oct 30 19:54:40.163901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.163851 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:40.498550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.498234 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:40.500566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.500536 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:40.500795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.500764 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:40.634175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.634136 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw"] Oct 30 19:54:40.635034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.635009 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b7855042-e67b-441c-b7b9-49e47aea7015" containerName="revoke-keys" Oct 30 19:54:40.635034 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.635033 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7855042-e67b-441c-b7b9-49e47aea7015" containerName="revoke-keys" Oct 30 19:54:40.635308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.635292 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7855042-e67b-441c-b7b9-49e47aea7015" containerName="revoke-keys" Oct 30 19:54:40.639346 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.639321 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.653077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.653047 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:40.653332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.653280 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:54:40.658526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.657142 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw"] Oct 30 19:54:40.745172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.745128 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.745381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.745227 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pp65\" (UniqueName: \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.745381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.745261 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.838656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.838616 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" event={"ID":"6135aae8-d2e3-41b6-860b-50a7c13fbd7d","Type":"ContainerStarted","Data":"8e831a913bc685ba3b2705c0b65234aca8244f854829fd2e3a187cda3228f2ca"} Oct 30 19:54:40.838875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.838686 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" event={"ID":"6135aae8-d2e3-41b6-860b-50a7c13fbd7d","Type":"ContainerStarted","Data":"4ee4c3cc505a9f36845350442d677fe86d2407f62cc12f9278d29977fddb9152"} Oct 30 19:54:40.838875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.838718 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:40.840742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.840701 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" event={"ID":"6d33c656-b26e-429c-bbc2-2f76974ffd11","Type":"ContainerStarted","Data":"809bcfc6ce09e1587ff59f00f5eb4b9cad0222e152b24de451fe9dcf6b11f693"} Oct 30 19:54:40.840873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.840758 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" event={"ID":"6d33c656-b26e-429c-bbc2-2f76974ffd11","Type":"ContainerStarted","Data":"ffb6c44986446845a39cdc805f23e19e9ea0a1583299087c64dd13042af978c6"} Oct 30 19:54:40.840977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.840956 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:40.846693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.846661 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.846866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.846740 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pp65\" (UniqueName: \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.846866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.846770 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.847636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.847604 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.849225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.849203 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.863996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.863966 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pp65\" (UniqueName: \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:40.936726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.936666 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" podStartSLOduration=2.936649624 podStartE2EDuration="2.936649624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:40.896541901 +0000 UTC m=+2988.608067270" watchObservedRunningTime="2026-10-30 19:54:40.936649624 +0000 UTC m=+2988.648174989" Oct 30 19:54:40.937226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.937198 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" podStartSLOduration=2.93719074 podStartE2EDuration="2.93719074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:40.933220114 +0000 UTC m=+2988.644745502" watchObservedRunningTime="2026-10-30 19:54:40.93719074 +0000 UTC m=+2988.648716113" Oct 30 19:54:40.955249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:40.955214 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:41.024290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.024251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:41.308159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.308076 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw"] Oct 30 19:54:41.309012 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:41.308983 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37c370c_e79c_41e2_8b68_46f5822dd8be.slice/crio-782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36 WatchSource:0}: Error finding container 782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36: Status 404 returned error can't find the container with id 782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36 Oct 30 19:54:41.425629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.425583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:41.511216 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.511166 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:41.511698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.511231 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:41.588807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.588761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:41.749600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.749560 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:41.855235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.855191 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" event={"ID":"d37c370c-e79c-41e2-8b68-46f5822dd8be","Type":"ContainerStarted","Data":"782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36"} Oct 30 19:54:41.924768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:41.924728 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.422116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.419342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.519114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.517534 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:42.519114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.517593 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:42.747132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.744043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.895131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.894401 4990 generic.go:356] "Generic (PLEG): container finished" podID="d37c370c-e79c-41e2-8b68-46f5822dd8be" containerID="f7510a936621af5a534a2aacda97585cfdec0a27b8316b2a1d9c7789f3d5d63f" exitCode=0 Oct 30 19:54:42.895131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:42.894485 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" event={"ID":"d37c370c-e79c-41e2-8b68-46f5822dd8be","Type":"ContainerDied","Data":"f7510a936621af5a534a2aacda97585cfdec0a27b8316b2a1d9c7789f3d5d63f"} Oct 30 19:54:43.507881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:43.504473 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:43.507881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:43.505118 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:43.507881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:43.505192 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:44.210678 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.210609 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:44.306603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.306564 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\" (UniqueName: \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\") pod \"d37c370c-e79c-41e2-8b68-46f5822dd8be\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " Oct 30 19:54:44.307005 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.306979 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\") pod \"d37c370c-e79c-41e2-8b68-46f5822dd8be\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " Oct 30 19:54:44.307317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.307295 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\") pod \"d37c370c-e79c-41e2-8b68-46f5822dd8be\" (UID: \"d37c370c-e79c-41e2-8b68-46f5822dd8be\") " Oct 30 19:54:44.307717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.307653 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca" pod "d37c370c-e79c-41e2-8b68-46f5822dd8be" (UID: "d37c370c-e79c-41e2-8b68-46f5822dd8be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:44.308526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.308500 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:44.309282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.309256 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca" pod "d37c370c-e79c-41e2-8b68-46f5822dd8be" (UID: "d37c370c-e79c-41e2-8b68-46f5822dd8be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:44.309376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.309301 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65" pod "d37c370c-e79c-41e2-8b68-46f5822dd8be" (UID: "d37c370c-e79c-41e2-8b68-46f5822dd8be"). InnerVolumeSpecName "kube-api-access-8pp65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:44.410498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.410461 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c370c-e79c-41e2-8b68-46f5822dd8be-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:44.410498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.410497 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pp65\" (UniqueName: \"kubernetes.io/projected/d37c370c-e79c-41e2-8b68-46f5822dd8be-kube-api-access-8pp65\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:44.497988 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.497892 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:44.498220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.497975 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:44.990117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.983868 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" event={"ID":"d37c370c-e79c-41e2-8b68-46f5822dd8be","Type":"ContainerDied","Data":"782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36"} Oct 30 19:54:44.990117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.983939 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782f07bd236e6e09380b91da4e6e1a240f5b4dd1be1e04219797960f7864ae36" Oct 30 19:54:44.990117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:44.984021 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw" Oct 30 19:54:45.498124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.497858 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 19:54:45.498124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.497964 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 19:54:45.593662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.593619 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw"] Oct 30 19:54:45.595184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.595150 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d37c370c-e79c-41e2-8b68-46f5822dd8be" containerName="revoke-keys" Oct 30 19:54:45.595184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.595183 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37c370c-e79c-41e2-8b68-46f5822dd8be" containerName="revoke-keys" Oct 30 19:54:45.595492 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.595475 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37c370c-e79c-41e2-8b68-46f5822dd8be" containerName="revoke-keys" Oct 30 19:54:45.600624 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.600596 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.613987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.613942 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:45.615915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.615877 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw"] Oct 30 19:54:45.623991 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.623946 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:54:45.638296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.638244 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.638515 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.638486 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.638719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.638694 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm4g\" (UniqueName: \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.740698 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.740661 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.740939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.740805 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.741013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.740966 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm4g\" (UniqueName: \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.741608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.741584 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.743293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.743262 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.759166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.759054 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm4g\" (UniqueName: \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\") pod \"maas-api-revoke-sub-9f7efe73-qlkkw\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:45.926390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.926350 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.932845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:45.932805 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:46.118925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.118876 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.271804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.271735 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw"] Oct 30 19:54:46.273246 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:46.273214 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bccd00_d3d4_427e_9521_fb178021daa6.slice/crio-c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49 WatchSource:0}: Error finding container c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49: Status 404 returned error can't find the container with id c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49 Oct 30 19:54:46.278204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.278171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.503820 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.503783 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:46.504383 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.504249 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:46.504999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.504978 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 19:54:46.953275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:46.952749 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.015523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:47.015474 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" event={"ID":"b7bccd00-d3d4-427e-9521-fb178021daa6","Type":"ContainerStarted","Data":"ca22388381d8a88feeb81fbcd135a8c4379cde487faaace02b5f68e9d9f00c77"} Oct 30 19:54:47.015733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:47.015591 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" event={"ID":"b7bccd00-d3d4-427e-9521-fb178021daa6","Type":"ContainerStarted","Data":"c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49"} Oct 30 19:54:47.065305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:47.065230 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" podStartSLOduration=2.065207317 podStartE2EDuration="2.065207317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:47.061469484 +0000 UTC m=+2994.772994850" watchObservedRunningTime="2026-10-30 19:54:47.065207317 +0000 UTC m=+2994.776732685" Oct 30 19:54:47.835398 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:47.835312 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.997265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:47.997222 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.029637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:48.029078 4990 generic.go:356] "Generic (PLEG): container finished" podID="b7bccd00-d3d4-427e-9521-fb178021daa6" containerID="ca22388381d8a88feeb81fbcd135a8c4379cde487faaace02b5f68e9d9f00c77" exitCode=0 Oct 30 19:54:48.029637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:48.029234 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" event={"ID":"b7bccd00-d3d4-427e-9521-fb178021daa6","Type":"ContainerDied","Data":"ca22388381d8a88feeb81fbcd135a8c4379cde487faaace02b5f68e9d9f00c77"} Oct 30 19:54:48.843123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:48.838683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:49.231743 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.231712 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:49.337146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.337105 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\" (UniqueName: \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\") pod \"b7bccd00-d3d4-427e-9521-fb178021daa6\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " Oct 30 19:54:49.337328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.337242 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\") pod \"b7bccd00-d3d4-427e-9521-fb178021daa6\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " Oct 30 19:54:49.337328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.337313 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\") pod \"b7bccd00-d3d4-427e-9521-fb178021daa6\" (UID: \"b7bccd00-d3d4-427e-9521-fb178021daa6\") " Oct 30 19:54:49.337708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.337676 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca" pod "b7bccd00-d3d4-427e-9521-fb178021daa6" (UID: "b7bccd00-d3d4-427e-9521-fb178021daa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:49.338064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.338043 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:49.339593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.339562 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca" pod "b7bccd00-d3d4-427e-9521-fb178021daa6" (UID: "b7bccd00-d3d4-427e-9521-fb178021daa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:49.339697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.339632 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g" pod "b7bccd00-d3d4-427e-9521-fb178021daa6" (UID: "b7bccd00-d3d4-427e-9521-fb178021daa6"). InnerVolumeSpecName "kube-api-access-ccm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:49.439890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.439805 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccm4g\" (UniqueName: \"kubernetes.io/projected/b7bccd00-d3d4-427e-9521-fb178021daa6-kube-api-access-ccm4g\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:49.439890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:49.439838 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7bccd00-d3d4-427e-9521-fb178021daa6-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:54:50.049174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:50.049141 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" Oct 30 19:54:50.049600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:50.049142 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw" event={"ID":"b7bccd00-d3d4-427e-9521-fb178021daa6","Type":"ContainerDied","Data":"c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49"} Oct 30 19:54:50.049600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:50.049277 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30897ac1f421a0b6b82bd4052ef521baf7c30920e2d898875a12ead15170f49" Oct 30 19:54:50.423253 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:50.423013 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:51.127053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.127012 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:54:51.158414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.158376 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:54:51.311873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.311834 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:51.454361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.454267 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:51.878114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.877393 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:54:51.878493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:51.878466 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:54:52.251562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:52.251456 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.087762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:54.087615 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.576396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:54.576361 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:54:54.587995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:54.587961 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:54:54.608904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:54.608870 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:54:54.618537 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:54.618513 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:54:55.690009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:55.689970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.127801 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.127757 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" containerID="cri-o://e7d497b187585227b3167f035848ac441d238622ee4f8974f4a8c13e21ee2af5" gracePeriod=25 Oct 30 19:54:56.158832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.158787 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" containerID="cri-o://400d1e28c4a453370d8ce65d469515ff45742230ebd08d9f2f76caa7ac78161e" gracePeriod=25 Oct 30 19:54:56.485818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.485717 4990 scope.go:122] "RemoveContainer" containerID="cf21b7f3dded6f4008f5285072580f0b0ddae22c49a51b10fbcb90ff7fd668bc" Oct 30 19:54:56.521637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.521602 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw"] Oct 30 19:54:56.522873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.522852 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b7bccd00-d3d4-427e-9521-fb178021daa6" containerName="revoke-keys" Oct 30 19:54:56.522920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.522877 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bccd00-d3d4-427e-9521-fb178021daa6" containerName="revoke-keys" Oct 30 19:54:56.523203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.523181 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bccd00-d3d4-427e-9521-fb178021daa6" containerName="revoke-keys" Oct 30 19:54:56.528356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.528340 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.543825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.543797 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw"] Oct 30 19:54:56.545974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.545937 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:56.546076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.546054 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:54:56.560288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.558908 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hg5\" (UniqueName: \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.560288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.559001 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.560288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.559213 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.660637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.660599 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hg5\" (UniqueName: \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.660845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.660675 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.660845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.660743 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.661412 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.661385 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.663162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.663140 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.678741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.678704 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hg5\" (UniqueName: \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\") pod \"maas-api-revoke-sub-dcb8362b-29tmw\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:56.847030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:56.846989 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:54:57.171877 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:57.171791 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw"] Oct 30 19:54:57.176520 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:54:57.176482 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5162479c_c704_4965_959c_fcc13f9e2259.slice/crio-a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19 WatchSource:0}: Error finding container a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19: Status 404 returned error can't find the container with id a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19 Oct 30 19:54:57.338390 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:57.333208 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:58.137481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:58.137441 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" event={"ID":"5162479c-c704-4965-959c-fcc13f9e2259","Type":"ContainerStarted","Data":"bae7aac276e7b8c2f329b2729e07da16d522eaf4a1b2882b6e2d321fdf3dc4e2"} Oct 30 19:54:58.137966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:58.137509 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" event={"ID":"5162479c-c704-4965-959c-fcc13f9e2259","Type":"ContainerStarted","Data":"a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19"} Oct 30 19:54:58.175952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:58.175886 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" podStartSLOduration=2.17587157 podStartE2EDuration="2.17587157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:58.171064668 +0000 UTC m=+3005.882590034" watchObservedRunningTime="2026-10-30 19:54:58.17587157 +0000 UTC m=+3005.887396967" Oct 30 19:54:59.149078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:59.149044 4990 generic.go:356] "Generic (PLEG): container finished" podID="5162479c-c704-4965-959c-fcc13f9e2259" containerID="bae7aac276e7b8c2f329b2729e07da16d522eaf4a1b2882b6e2d321fdf3dc4e2" exitCode=0 Oct 30 19:54:59.149685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:59.149128 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" event={"ID":"5162479c-c704-4965-959c-fcc13f9e2259","Type":"ContainerDied","Data":"bae7aac276e7b8c2f329b2729e07da16d522eaf4a1b2882b6e2d321fdf3dc4e2"} Oct 30 19:54:59.270327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:59.270287 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:59.415620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:59.415522 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:59.555268 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:54:59.555235 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.348177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.348146 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:55:00.427147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.427020 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\" (UniqueName: \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\") pod \"5162479c-c704-4965-959c-fcc13f9e2259\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " Oct 30 19:55:00.427586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.427555 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\") pod \"5162479c-c704-4965-959c-fcc13f9e2259\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " Oct 30 19:55:00.427721 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.427614 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\") pod \"5162479c-c704-4965-959c-fcc13f9e2259\" (UID: \"5162479c-c704-4965-959c-fcc13f9e2259\") " Oct 30 19:55:00.427997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.427963 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca" pod "5162479c-c704-4965-959c-fcc13f9e2259" (UID: "5162479c-c704-4965-959c-fcc13f9e2259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:00.428506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.428482 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:00.429947 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.429922 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5" pod "5162479c-c704-4965-959c-fcc13f9e2259" (UID: "5162479c-c704-4965-959c-fcc13f9e2259"). InnerVolumeSpecName "kube-api-access-29hg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:00.430207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.430170 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca" pod "5162479c-c704-4965-959c-fcc13f9e2259" (UID: "5162479c-c704-4965-959c-fcc13f9e2259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:00.548154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.539387 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hg5\" (UniqueName: \"kubernetes.io/projected/5162479c-c704-4965-959c-fcc13f9e2259-kube-api-access-29hg5\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:00.548154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.539444 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5162479c-c704-4965-959c-fcc13f9e2259-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:00.627517 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.627447 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:00.629178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:00.629086 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:01.178671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:01.178639 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" Oct 30 19:55:01.178879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:01.178672 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw" event={"ID":"5162479c-c704-4965-959c-fcc13f9e2259","Type":"ContainerDied","Data":"a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19"} Oct 30 19:55:01.178879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:01.178720 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ade106a679d5f75435e596430adcf79c03029965faf439db21306eab4def19" Oct 30 19:55:01.255407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:01.255369 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.401077 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:02.401039 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.534044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:03.534009 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.102581 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:55:05.102541 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:55:05.280585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.280530 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.785423 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.785391 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9"] Oct 30 19:55:05.786232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.786208 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5162479c-c704-4965-959c-fcc13f9e2259" containerName="revoke-keys" Oct 30 19:55:05.786232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.786225 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5162479c-c704-4965-959c-fcc13f9e2259" containerName="revoke-keys" Oct 30 19:55:05.786377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.786368 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5162479c-c704-4965-959c-fcc13f9e2259" containerName="revoke-keys" Oct 30 19:55:05.790493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.790474 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.810321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.810290 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9"] Oct 30 19:55:05.811957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.811924 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:05.812072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.811986 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:55:05.860766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.860731 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.861311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.861286 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.861608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.861589 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjcl\" (UniqueName: \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.963329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.963296 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.963493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.963398 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjcl\" (UniqueName: \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.963539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.963494 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.964264 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.964235 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.965827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.965807 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:05.980486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:05.980462 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjcl\" (UniqueName: \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\") pod \"maas-api-revoke-sub-d7e5c5a3-4bjq9\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:06.107580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:06.107549 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:06.428649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:06.428573 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9"] Oct 30 19:55:06.432806 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:06.432765 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aacf09c_465b_4e69_bf1f_59c3fe19370b.slice/crio-289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e WatchSource:0}: Error finding container 289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e: Status 404 returned error can't find the container with id 289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e Oct 30 19:55:06.982598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:06.982570 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk"] Oct 30 19:55:06.987469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:06.987450 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.005957 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.005922 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk"] Oct 30 19:55:07.058733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.050015 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.082495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.082463 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nwn\" (UniqueName: \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.082712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.082684 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.082776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.082756 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.184699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.184660 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.185225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.184719 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.185225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.184935 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nwn\" (UniqueName: \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.185485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.185462 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.187044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.187020 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.202661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.202624 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nwn\" (UniqueName: \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\") pod \"maas-api-revoke-sub-2afc22d0-58jfk\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.283825 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.283786 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" event={"ID":"5aacf09c-465b-4e69-bf1f-59c3fe19370b","Type":"ContainerStarted","Data":"9f7d82932e2989b0a3fa7076874267f2b5bc2614287ec656cc2328676c46603c"} Oct 30 19:55:07.284011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.283848 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" event={"ID":"5aacf09c-465b-4e69-bf1f-59c3fe19370b","Type":"ContainerStarted","Data":"289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e"} Oct 30 19:55:07.306324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.306295 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:07.319508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.319435 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" podStartSLOduration=2.319414991 podStartE2EDuration="2.319414991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:07.317374284 +0000 UTC m=+3015.028899664" watchObservedRunningTime="2026-10-30 19:55:07.319414991 +0000 UTC m=+3015.030940358" Oct 30 19:55:07.658970 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:07.658923 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0ef85f_7a36_4caa_8a9a_6c47313a4e38.slice/crio-46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4 WatchSource:0}: Error finding container 46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4: Status 404 returned error can't find the container with id 46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4 Oct 30 19:55:07.660043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:07.660019 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk"] Oct 30 19:55:08.296876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.296780 4990 generic.go:356] "Generic (PLEG): container finished" podID="5aacf09c-465b-4e69-bf1f-59c3fe19370b" containerID="9f7d82932e2989b0a3fa7076874267f2b5bc2614287ec656cc2328676c46603c" exitCode=0 Oct 30 19:55:08.296876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.296857 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" event={"ID":"5aacf09c-465b-4e69-bf1f-59c3fe19370b","Type":"ContainerDied","Data":"9f7d82932e2989b0a3fa7076874267f2b5bc2614287ec656cc2328676c46603c"} Oct 30 19:55:08.298739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.298716 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" event={"ID":"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38","Type":"ContainerStarted","Data":"7a80e1c180c5513d74210c04cfe8a772d7559962c96a07e7ee74099ab57c2f38"} Oct 30 19:55:08.298860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.298759 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" event={"ID":"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38","Type":"ContainerStarted","Data":"46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4"} Oct 30 19:55:08.372018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.371955 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" podStartSLOduration=2.371935961 podStartE2EDuration="2.371935961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:08.36423988 +0000 UTC m=+3016.075765246" watchObservedRunningTime="2026-10-30 19:55:08.371935961 +0000 UTC m=+3016.083461329" Oct 30 19:55:08.636493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:08.636452 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.308859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.308823 4990 generic.go:356] "Generic (PLEG): container finished" podID="3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" containerID="7a80e1c180c5513d74210c04cfe8a772d7559962c96a07e7ee74099ab57c2f38" exitCode=0 Oct 30 19:55:09.309395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.308908 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" event={"ID":"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38","Type":"ContainerDied","Data":"7a80e1c180c5513d74210c04cfe8a772d7559962c96a07e7ee74099ab57c2f38"} Oct 30 19:55:09.493217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.493190 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:09.630851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.630773 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\") pod \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " Oct 30 19:55:09.631009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.630937 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\") pod \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " Oct 30 19:55:09.631159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.631144 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\" (UniqueName: \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\") pod \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\" (UID: \"5aacf09c-465b-4e69-bf1f-59c3fe19370b\") " Oct 30 19:55:09.631159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.631146 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca" pod "5aacf09c-465b-4e69-bf1f-59c3fe19370b" (UID: "5aacf09c-465b-4e69-bf1f-59c3fe19370b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:09.631621 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.631603 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:09.633056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.633029 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca" pod "5aacf09c-465b-4e69-bf1f-59c3fe19370b" (UID: "5aacf09c-465b-4e69-bf1f-59c3fe19370b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:09.633254 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.633230 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl" pod "5aacf09c-465b-4e69-bf1f-59c3fe19370b" (UID: "5aacf09c-465b-4e69-bf1f-59c3fe19370b"). InnerVolumeSpecName "kube-api-access-9fjcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:09.732804 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.732769 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aacf09c-465b-4e69-bf1f-59c3fe19370b-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:09.732936 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:09.732811 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjcl\" (UniqueName: \"kubernetes.io/projected/5aacf09c-465b-4e69-bf1f-59c3fe19370b-kube-api-access-9fjcl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:10.325458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.325421 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.333300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.330988 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" Oct 30 19:55:10.341601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.341568 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9" event={"ID":"5aacf09c-465b-4e69-bf1f-59c3fe19370b","Type":"ContainerDied","Data":"289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e"} Oct 30 19:55:10.341814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.341622 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="289a0865f5d7b6ca4c9db7335559ea72a13cc81d18388cec91bd07a21366b51e" Oct 30 19:55:10.531742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.531712 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:10.624565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.624468 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:10.624733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.624607 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:10.648313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.648287 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\" (UniqueName: \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\") pod \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " Oct 30 19:55:10.648488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.648411 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\") pod \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " Oct 30 19:55:10.648488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.648443 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\") pod \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\" (UID: \"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38\") " Oct 30 19:55:10.648780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.648754 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca" pod "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" (UID: "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:10.649274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.649254 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:10.650688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.650656 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn" pod "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" (UID: "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38"). InnerVolumeSpecName "kube-api-access-m2nwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:10.650688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.650669 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca" pod "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" (UID: "3f0ef85f-7a36-4caa-8a9a-6c47313a4e38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:10.750760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.750723 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nwn\" (UniqueName: \"kubernetes.io/projected/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-kube-api-access-m2nwn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:10.750760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:10.750757 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:11.342557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:11.342525 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" Oct 30 19:55:11.342979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:11.342568 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk" event={"ID":"3f0ef85f-7a36-4caa-8a9a-6c47313a4e38","Type":"ContainerDied","Data":"46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4"} Oct 30 19:55:11.342979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:11.342628 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f084d0484d4ca12cf354b417299e5878dac89f74fa6adc3abd31db51f0a8b4" Oct 30 19:55:11.961999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:11.961957 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.325531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:13.325498 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.924900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:13.924865 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:14.016232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:14.016200 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw"] Oct 30 19:55:14.024872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:14.024845 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a7b7bc-dr7pw"] Oct 30 19:55:14.080679 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:14.080642 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:14.406213 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:14.406173 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37c370c-e79c-41e2-8b68-46f5822dd8be" path="/var/lib/kubelet/pods/d37c370c-e79c-41e2-8b68-46f5822dd8be/volumes" Oct 30 19:55:14.548356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:14.548314 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:15.836420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:15.836383 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:16.128472 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.128397 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr"] Oct 30 19:55:16.129243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129226 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5aacf09c-465b-4e69-bf1f-59c3fe19370b" containerName="revoke-keys" Oct 30 19:55:16.129340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129243 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aacf09c-465b-4e69-bf1f-59c3fe19370b" containerName="revoke-keys" Oct 30 19:55:16.129340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129259 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" containerName="revoke-keys" Oct 30 19:55:16.129340 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129264 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" containerName="revoke-keys" Oct 30 19:55:16.129440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129376 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aacf09c-465b-4e69-bf1f-59c3fe19370b" containerName="revoke-keys" Oct 30 19:55:16.129440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.129388 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" containerName="revoke-keys" Oct 30 19:55:16.134704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.134681 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.150507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.150480 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.150841 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.150817 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.150952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.150865 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t99\" (UniqueName: \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.151017 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.150952 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr"] Oct 30 19:55:16.153649 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.153625 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:16.153737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.153685 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:55:16.251973 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.251941 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.252186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.251987 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t99\" (UniqueName: \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.252186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.252064 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.252846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.252818 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.254526 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.254497 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.270266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.270233 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t99\" (UniqueName: \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\") pod \"maas-api-revoke-sub-caa49937-pxqkr\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.305637 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.305413 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:16.406330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.406040 4990 generic.go:356] "Generic (PLEG): container finished" podID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerID="400d1e28c4a453370d8ce65d469515ff45742230ebd08d9f2f76caa7ac78161e" exitCode=0 Oct 30 19:55:16.408753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.408644 4990 generic.go:356] "Generic (PLEG): container finished" podID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerID="e7d497b187585227b3167f035848ac441d238622ee4f8974f4a8c13e21ee2af5" exitCode=0 Oct 30 19:55:16.419333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.418912 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" event={"ID":"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8","Type":"ContainerDied","Data":"400d1e28c4a453370d8ce65d469515ff45742230ebd08d9f2f76caa7ac78161e"} Oct 30 19:55:16.419333 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.418968 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" event={"ID":"e5e339a2-eb62-4b7a-a59f-3ddee83607eb","Type":"ContainerDied","Data":"e7d497b187585227b3167f035848ac441d238622ee4f8974f4a8c13e21ee2af5"} Oct 30 19:55:16.464290 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.463743 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:55:16.545982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.545954 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:55:16.566464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.566434 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\") pod \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " Oct 30 19:55:16.566635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.566540 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\" (UniqueName: \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\") pod \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\" (UID: \"e5e339a2-eb62-4b7a-a59f-3ddee83607eb\") " Oct 30 19:55:16.566789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.566757 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\") pod \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " Oct 30 19:55:16.566927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.566812 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\" (UniqueName: \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\") pod \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\" (UID: \"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8\") " Oct 30 19:55:16.567015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.566950 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume" pod "e5e339a2-eb62-4b7a-a59f-3ddee83607eb" (UID: "e5e339a2-eb62-4b7a-a59f-3ddee83607eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:16.567396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.567373 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume" pod "ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" (UID: "ae035cbe-7c51-4b01-b16e-f2ffdca3abd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:16.567522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.567506 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:16.567591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.567527 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:16.569840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.569813 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng" pod "ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" (UID: "ae035cbe-7c51-4b01-b16e-f2ffdca3abd8"). InnerVolumeSpecName "kube-api-access-jb2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:16.570260 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.570233 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p" pod "e5e339a2-eb62-4b7a-a59f-3ddee83607eb" (UID: "e5e339a2-eb62-4b7a-a59f-3ddee83607eb"). InnerVolumeSpecName "kube-api-access-2xp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:16.668473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.668442 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr"] Oct 30 19:55:16.669192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.669167 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2ng\" (UniqueName: \"kubernetes.io/projected/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8-kube-api-access-jb2ng\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:16.669298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:16.669195 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xp6p\" (UniqueName: \"kubernetes.io/projected/e5e339a2-eb62-4b7a-a59f-3ddee83607eb-kube-api-access-2xp6p\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:16.669742 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:16.669721 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8c3f45_8ad0_46f0_9ef3_ef534660f91e.slice/crio-9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f WatchSource:0}: Error finding container 9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f: Status 404 returned error can't find the container with id 9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f Oct 30 19:55:17.422251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.422147 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" event={"ID":"e5e339a2-eb62-4b7a-a59f-3ddee83607eb","Type":"ContainerDied","Data":"ff57b24f6c7faa11380254d1b7541c8a81fa090e71ab208b9e91df5cc948f712"} Oct 30 19:55:17.422251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.422164 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx" Oct 30 19:55:17.423555 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.422216 4990 scope.go:122] "RemoveContainer" containerID="e7d497b187585227b3167f035848ac441d238622ee4f8974f4a8c13e21ee2af5" Oct 30 19:55:17.428235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.428204 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" event={"ID":"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e","Type":"ContainerStarted","Data":"1850af562e5fa039c061e8b8f958b2277361cea2533792ce6b407972a8cee272"} Oct 30 19:55:17.428402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.428259 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" event={"ID":"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e","Type":"ContainerStarted","Data":"9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f"} Oct 30 19:55:17.430424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.430404 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" Oct 30 19:55:17.430557 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.430438 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz" event={"ID":"ae035cbe-7c51-4b01-b16e-f2ffdca3abd8","Type":"ContainerDied","Data":"574f711d21fca48f6182d27adc3ecd64ef9fd52238b8cabfcd7747d239ccd53f"} Oct 30 19:55:17.440590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.440572 4990 scope.go:122] "RemoveContainer" containerID="400d1e28c4a453370d8ce65d469515ff45742230ebd08d9f2f76caa7ac78161e" Oct 30 19:55:17.486023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.485958 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" podStartSLOduration=1.485942066 podStartE2EDuration="1.485942066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:17.484827209 +0000 UTC m=+3025.196352574" watchObservedRunningTime="2026-10-30 19:55:17.485942066 +0000 UTC m=+3025.197467433" Oct 30 19:55:17.520967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.520926 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:55:17.526231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.526204 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a7b7bc-867945d5cd-v6bsx"] Oct 30 19:55:17.547948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.547251 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:17.560680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.560653 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:55:17.576148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.571485 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a7b7bc-6c6879749f-m6dwz"] Oct 30 19:55:17.940646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:17.940496 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:55:18.011868 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:18.011832 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:18.405863 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:18.405824 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" path="/var/lib/kubelet/pods/ae035cbe-7c51-4b01-b16e-f2ffdca3abd8/volumes" Oct 30 19:55:18.406662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:18.406639 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" path="/var/lib/kubelet/pods/e5e339a2-eb62-4b7a-a59f-3ddee83607eb/volumes" Oct 30 19:55:18.441527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:18.441496 4990 generic.go:356] "Generic (PLEG): container finished" podID="8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" containerID="1850af562e5fa039c061e8b8f958b2277361cea2533792ce6b407972a8cee272" exitCode=0 Oct 30 19:55:18.441918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:18.441572 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" event={"ID":"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e","Type":"ContainerDied","Data":"1850af562e5fa039c061e8b8f958b2277361cea2533792ce6b407972a8cee272"} Oct 30 19:55:19.130006 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.129944 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.636240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.636215 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:19.750523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.750441 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\" (UniqueName: \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\") pod \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " Oct 30 19:55:19.750712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.750698 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\") pod \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " Oct 30 19:55:19.750765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.750740 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\") pod \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\" (UID: \"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e\") " Oct 30 19:55:19.751248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.751222 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca" pod "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" (UID: "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:19.751509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.751492 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:19.752627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.752590 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99" pod "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" (UID: "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e"). InnerVolumeSpecName "kube-api-access-l7t99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:19.752727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.752671 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca" pod "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" (UID: "8a8c3f45-8ad0-46f0-9ef3-ef534660f91e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:19.852861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.852833 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:19.852861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:19.852858 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t99\" (UniqueName: \"kubernetes.io/projected/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e-kube-api-access-l7t99\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:20.461559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:20.461532 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" Oct 30 19:55:20.461741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:20.461572 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr" event={"ID":"8a8c3f45-8ad0-46f0-9ef3-ef534660f91e","Type":"ContainerDied","Data":"9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f"} Oct 30 19:55:20.461741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:20.461626 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9adf6b16b21c0614692798744a93785a87f568ad8640493d458a6099c8b4c56f" Oct 30 19:55:20.681173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:20.670722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:20.998059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:20.997976 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:21.048285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:21.048253 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw"] Oct 30 19:55:21.055900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:21.055873 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a132ac6a-v67xw"] Oct 30 19:55:21.187451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:21.187417 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:22.407307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:22.407270 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9d1b39-e2d2-41bf-a5b9-625922d0115c" path="/var/lib/kubelet/pods/1b9d1b39-e2d2-41bf-a5b9-625922d0115c/volumes" Oct 30 19:55:23.062912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:23.060614 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.286961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.286926 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.459085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.452163 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.711158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.710146 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:55:24.711376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711345 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" Oct 30 19:55:24.711376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711367 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" Oct 30 19:55:24.711497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711413 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" containerName="revoke-keys" Oct 30 19:55:24.711497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711420 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" containerName="revoke-keys" Oct 30 19:55:24.711497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711442 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" Oct 30 19:55:24.711497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711450 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" Oct 30 19:55:24.711725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711637 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e339a2-eb62-4b7a-a59f-3ddee83607eb" containerName="payload-processing" Oct 30 19:55:24.711725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711656 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" containerName="revoke-keys" Oct 30 19:55:24.711725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.711666 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae035cbe-7c51-4b01-b16e-f2ffdca3abd8" containerName="payload-pre-processing" Oct 30 19:55:24.728114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.716123 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:24.746912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.741901 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b36ad-dockercfg-vg5nk\"" Oct 30 19:55:24.746912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.742660 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b36ad\"" Oct 30 19:55:24.751702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.749223 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:55:24.885189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.883550 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:24.885189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.883793 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhmp\" (UniqueName: \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:24.984873 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.984833 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhmp\" (UniqueName: \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:24.985084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.984946 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:24.985929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:24.985899 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:25.002035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:25.002002 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhmp\" (UniqueName: \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:25.149942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:25.149855 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:25.438234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:25.436801 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:25.472438 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:25.472406 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33c140d_ef70_4b89_91cd_d5bb0078a3a6.slice/crio-6c0757a65aed04249ff628ae7666ad79603371226725aa65f909059a661acf05 WatchSource:0}: Error finding container 6c0757a65aed04249ff628ae7666ad79603371226725aa65f909059a661acf05: Status 404 returned error can't find the container with id 6c0757a65aed04249ff628ae7666ad79603371226725aa65f909059a661acf05 Oct 30 19:55:25.474575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:25.474546 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:55:25.517477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:25.517445 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" event={"ID":"e33c140d-ef70-4b89-91cd-d5bb0078a3a6","Type":"ContainerStarted","Data":"6c0757a65aed04249ff628ae7666ad79603371226725aa65f909059a661acf05"} Oct 30 19:55:26.527366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:26.527328 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" event={"ID":"e33c140d-ef70-4b89-91cd-d5bb0078a3a6","Type":"ContainerStarted","Data":"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d"} Oct 30 19:55:26.527753 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:26.527382 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:27.323626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:27.323587 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:27.607106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:27.606980 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" podStartSLOduration=3.6069599180000003 podStartE2EDuration="3.606959918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:26.571564963 +0000 UTC m=+3034.283090330" watchObservedRunningTime="2026-10-30 19:55:27.606959918 +0000 UTC m=+3035.318485286" Oct 30 19:55:27.607592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:27.607492 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:55:27.668749 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:27.668704 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:28.303438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.303401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:28.666826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.666739 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 19:55:28.672183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.672158 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.710051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.709969 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-315ea2-openshift-default-dockercfg-wp4c7\"" Oct 30 19:55:28.715843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.715813 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 19:55:28.744934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.744895 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.744962 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745024 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745102 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745153 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6qn\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745185 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745262 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745394 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.745470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.745432 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.773571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.773539 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:28.880249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880131 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880249 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880206 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880265 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880328 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880375 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6qn\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880406 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880490 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880627 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.880896 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.880656 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.881527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.881497 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.889150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.881682 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.889656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.889631 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.904918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.893814 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.941316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.899959 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.941650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.935017 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:28.960113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.946807 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.960113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.947632 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.978119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.974452 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:28.992885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:28.985860 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6qn\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\") pod \"e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:29.004719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.004688 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:29.095459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.095395 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.463298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.463264 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 19:55:29.463608 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:29.463580 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9557ea28_d430_4305_b313_107486eb594b.slice/crio-3633274f3e78b506515cd4fef29589f991a1a5e36e1cfb33d717454879cd3d89 WatchSource:0}: Error finding container 3633274f3e78b506515cd4fef29589f991a1a5e36e1cfb33d717454879cd3d89: Status 404 returned error can't find the container with id 3633274f3e78b506515cd4fef29589f991a1a5e36e1cfb33d717454879cd3d89 Oct 30 19:55:29.466677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.466644 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:29.466765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.466720 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:29.466765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.466752 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:29.511347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.511315 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.594118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.593908 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerStarted","Data":"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081"} Oct 30 19:55:29.594118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.593968 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerStarted","Data":"3633274f3e78b506515cd4fef29589f991a1a5e36e1cfb33d717454879cd3d89"} Oct 30 19:55:29.634760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:29.634707 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podStartSLOduration=1.634693898 podStartE2EDuration="1.634693898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:29.631037588 +0000 UTC m=+3037.342562953" watchObservedRunningTime="2026-10-30 19:55:29.634693898 +0000 UTC m=+3037.346219263" Oct 30 19:55:30.011348 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.011299 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:30.018142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.018079 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:30.510239 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.510206 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:55:30.514662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.514643 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.534026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.533995 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-59f9ed-serving-cert\"" Oct 30 19:55:30.538583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.538555 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:55:30.605784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.605753 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:30.606858 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.606834 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:55:30.634639 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.634608 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:30.646592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.646564 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.646894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.646867 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2klm\" (UniqueName: \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.647023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.646938 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.647023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.647008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.749028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.748997 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2klm\" (UniqueName: \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.749214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.749056 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.749214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.749135 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.749321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.749235 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.749933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.749906 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.751845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.751816 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.752309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.752285 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.766207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.766150 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2klm\" (UniqueName: \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\") pod \"maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.839032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.838294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:30.840605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.840575 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:30.925118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.924075 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:55:30.925118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.924290 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" containerID="cri-o://84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c" gracePeriod=30 Oct 30 19:55:30.994112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:30.994033 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.250890 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.239838 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.259440 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.259411 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567"] Oct 30 19:55:31.259788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.259770 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:55:31.273155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.273128 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4d2b811-jp567"] Oct 30 19:55:31.278800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.278775 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:55:31.623442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:31.623412 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" event={"ID":"d2b4ce16-f26b-4fea-8adb-1826cb1497a5","Type":"ContainerStarted","Data":"d02c52cc0f70cd07ca97e4cf8924d4997828826eb65038df58eceff61a800e56"} Oct 30 19:55:32.408136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:32.408081 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf" path="/var/lib/kubelet/pods/0f0a5e5a-c2c3-4f67-bb11-594feca3a9cf/volumes" Oct 30 19:55:32.635154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:32.635114 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" event={"ID":"d2b4ce16-f26b-4fea-8adb-1826cb1497a5","Type":"ContainerStarted","Data":"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee"} Oct 30 19:55:32.673797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:32.673683 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" podStartSLOduration=2.333723361 podStartE2EDuration="2.673642266s" totalImagesPullingTime="339.918905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:32.668943541 +0000 UTC m=+3040.380468900" watchObservedRunningTime="2026-10-30 19:55:32.673642266 +0000 UTC m=+3040.385167632" Oct 30 19:55:32.833822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:32.826570 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.963513 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:32.963422 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerName="main" containerID="cri-o://7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7" gracePeriod=45 Oct 30 19:55:33.056822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.056784 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.236:8000/ready\": dial tcp 10.134.0.236:8000: connect: connection refused" Oct 30 19:55:33.250113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.250076 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:55:33.313401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313371 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.313561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313478 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\" (UniqueName: \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.313561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313545 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\" (UniqueName: \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.313762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313734 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.313880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313810 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.314081 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.313999 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:33.314199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.314135 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\") pod \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\" (UID: \"566631b8-0dd8-4d0a-bae4-5c93e161fed7\") " Oct 30 19:55:33.314314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.314290 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:33.314482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.314456 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:33.315409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.315383 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.315510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.315414 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.315510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.315432 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.315977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.315955 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:33.315977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.315964 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "kube-api-access-pvnzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:33.317511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.317490 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs" pod "566631b8-0dd8-4d0a-bae4-5c93e161fed7" (UID: "566631b8-0dd8-4d0a-bae4-5c93e161fed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:33.416314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.416284 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566631b8-0dd8-4d0a-bae4-5c93e161fed7-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.416314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.416310 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnzv\" (UniqueName: \"kubernetes.io/projected/566631b8-0dd8-4d0a-bae4-5c93e161fed7-kube-api-access-pvnzv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.416704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.416321 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566631b8-0dd8-4d0a-bae4-5c93e161fed7-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:33.745116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.741120 4990 generic.go:356] "Generic (PLEG): container finished" podID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerID="7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7" exitCode=0 Oct 30 19:55:33.745116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.742773 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" Oct 30 19:55:33.745116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.744816 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" event={"ID":"566631b8-0dd8-4d0a-bae4-5c93e161fed7","Type":"ContainerDied","Data":"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7"} Oct 30 19:55:33.745116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.744867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd" event={"ID":"566631b8-0dd8-4d0a-bae4-5c93e161fed7","Type":"ContainerDied","Data":"c3ec9b3f958757a4f1aa225cb1071e9731ab880fe45f2bb5f5b7cab8620f8340"} Oct 30 19:55:33.745116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.744895 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:33.749110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.745600 4990 scope.go:122] "RemoveContainer" containerID="7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7" Oct 30 19:55:33.794839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.794809 4990 scope.go:122] "RemoveContainer" containerID="7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7" Oct 30 19:55:33.801146 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:55:33.796402 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7\": container with ID starting with 7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7 not found: ID does not exist" containerID="7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7" Oct 30 19:55:33.801146 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.796440 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7"} err="failed to get container status \"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7\": rpc error: code = NotFound desc = could not find container \"7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7\": container with ID starting with 7ec2b30f5846b1f32bcb1f0f03f3cbb0cfc6fc39bfdf75d5ddbf031bddc99ad7 not found: ID does not exist" Oct 30 19:55:33.842748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.841104 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:55:33.853153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:33.853124 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/sub-test-model-095f55-kserve-cd9999d85-p7pkd"] Oct 30 19:55:34.417141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:34.417062 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" path="/var/lib/kubelet/pods/566631b8-0dd8-4d0a-bae4-5c93e161fed7/volumes" Oct 30 19:55:34.606879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:34.606800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.097782 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:55:35.097747 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:55:36.311036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.310996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.666236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.666207 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:55:36.716230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716191 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716342 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716391 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716431 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716589 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716529 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716621 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.716692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716684 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.716942 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\") pod \"635573ca-0a7b-4a62-85d2-17ade59afb1f\" (UID: \"635573ca-0a7b-4a62-85d2-17ade59afb1f\") " Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717085 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717454 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717693 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717805 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717899 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717906 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717932 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.717982 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.717959 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/635573ca-0a7b-4a62-85d2-17ade59afb1f-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.718776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.718731 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:36.720946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.720050 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:36.720946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.720303 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "kube-api-access-j69k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:36.721145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.720966 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo" pod "635573ca-0a7b-4a62-85d2-17ade59afb1f" (UID: "635573ca-0a7b-4a62-85d2-17ade59afb1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:36.778587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.778550 4990 generic.go:356] "Generic (PLEG): container finished" podID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerID="84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c" exitCode=0 Oct 30 19:55:36.778827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.778617 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" Oct 30 19:55:36.778827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.778636 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerDied","Data":"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c"} Oct 30 19:55:36.778827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.778703 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz" event={"ID":"635573ca-0a7b-4a62-85d2-17ade59afb1f","Type":"ContainerDied","Data":"9164125e612f3c63521546240a5015f827ec5366bdd1c56bdc069c5a6ae1006b"} Oct 30 19:55:36.778827 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.778732 4990 scope.go:122] "RemoveContainer" containerID="84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c" Oct 30 19:55:36.794768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.794745 4990 scope.go:122] "RemoveContainer" containerID="d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60" Oct 30 19:55:36.806041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.806018 4990 scope.go:122] "RemoveContainer" containerID="84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c" Oct 30 19:55:36.806426 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:55:36.806388 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c\": container with ID starting with 84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c not found: ID does not exist" containerID="84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c" Oct 30 19:55:36.806538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.806437 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c"} err="failed to get container status \"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c\": rpc error: code = NotFound desc = could not find container \"84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c\": container with ID starting with 84504b024ed1dc0de8afc3c30fd974638e7081f98b1a5a57154968b631b06f7c not found: ID does not exist" Oct 30 19:55:36.806538 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.806459 4990 scope.go:122] "RemoveContainer" containerID="d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60" Oct 30 19:55:36.806773 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:55:36.806744 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60\": container with ID starting with d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60 not found: ID does not exist" containerID="d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60" Oct 30 19:55:36.806837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.806784 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60"} err="failed to get container status \"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60\": rpc error: code = NotFound desc = could not find container \"d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60\": container with ID starting with d17d6211f2363d7387d38cda2ab4ddee0dd5d418ecef73b65098f562af6dcd60 not found: ID does not exist" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820017 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820046 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820060 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820073 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820103 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69k6\" (UniqueName: \"kubernetes.io/projected/635573ca-0a7b-4a62-85d2-17ade59afb1f-kube-api-access-j69k6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.820945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.820118 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/635573ca-0a7b-4a62-85d2-17ade59afb1f-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:36.831968 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.831762 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:55:36.838824 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.837213 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7b7bc-openshift-default-848696d657-dqstz"] Oct 30 19:55:36.910205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.910133 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn"] Oct 30 19:55:36.911370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911345 4990 cpu_manager.go:414] "containerMap: removing container" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:36.911370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911370 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:36.911521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911386 4990 cpu_manager.go:414] "containerMap: removing container" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerName="main" Oct 30 19:55:36.911521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911394 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerName="main" Oct 30 19:55:36.911521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911451 4990 cpu_manager.go:414] "containerMap: removing container" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:36.911521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911458 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:36.911670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911656 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:36.911707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.911684 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="566631b8-0dd8-4d0a-bae4-5c93e161fed7" containerName="main" Oct 30 19:55:36.917945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.917796 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:36.931351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.931015 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:36.931351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.931171 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:55:36.932681 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:36.932656 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn"] Oct 30 19:55:37.023686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.023653 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.023867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.023782 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.024056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.024034 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnvz\" (UniqueName: \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.125458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.125417 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnvz\" (UniqueName: \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.125674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.125486 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.125674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.125604 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.126416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.126384 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.128776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.128744 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.142380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.142350 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnvz\" (UniqueName: \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\") pod \"maas-api-revoke-sub-693a1b3c-dgkqn\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.248041 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.247962 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:37.539135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.539038 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:55:37.569543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.569470 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn"] Oct 30 19:55:37.795722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:37.795636 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" event={"ID":"65b78e2c-855a-49e2-9de8-4fc5a8cfe984","Type":"ContainerStarted","Data":"b13a674bd8228956668c505c35a81da4a50f0d429234832773487baafa52ad79"} Oct 30 19:55:38.232111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:38.230329 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.408922 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:38.408885 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" path="/var/lib/kubelet/pods/635573ca-0a7b-4a62-85d2-17ade59afb1f/volumes" Oct 30 19:55:38.810932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:38.810893 4990 generic.go:356] "Generic (PLEG): container finished" podID="65b78e2c-855a-49e2-9de8-4fc5a8cfe984" containerID="74626cbfeab4db39bc473b8fe0979384dfff262116f930c43681ae2c6513b826" exitCode=0 Oct 30 19:55:38.811471 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:38.811215 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" event={"ID":"65b78e2c-855a-49e2-9de8-4fc5a8cfe984","Type":"ContainerDied","Data":"74626cbfeab4db39bc473b8fe0979384dfff262116f930c43681ae2c6513b826"} Oct 30 19:55:39.757650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:39.757621 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:55:40.019921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.019718 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.046933 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.046900 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:40.181066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.181036 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\") pod \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " Oct 30 19:55:40.181328 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.181311 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\" (UniqueName: \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\") pod \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " Oct 30 19:55:40.181409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.181355 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\") pod \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\" (UID: \"65b78e2c-855a-49e2-9de8-4fc5a8cfe984\") " Oct 30 19:55:40.181501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.181469 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca" pod "65b78e2c-855a-49e2-9de8-4fc5a8cfe984" (UID: "65b78e2c-855a-49e2-9de8-4fc5a8cfe984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:40.182242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.182187 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:40.184729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.184702 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca" pod "65b78e2c-855a-49e2-9de8-4fc5a8cfe984" (UID: "65b78e2c-855a-49e2-9de8-4fc5a8cfe984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:40.184893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.184868 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz" pod "65b78e2c-855a-49e2-9de8-4fc5a8cfe984" (UID: "65b78e2c-855a-49e2-9de8-4fc5a8cfe984"). InnerVolumeSpecName "kube-api-access-2tnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:40.283952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.283885 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tnvz\" (UniqueName: \"kubernetes.io/projected/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-kube-api-access-2tnvz\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:40.283952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.283912 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b78e2c-855a-49e2-9de8-4fc5a8cfe984-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:40.646832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.646793 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 19:55:40.647683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.647665 4990 cpu_manager.go:414] "containerMap: removing container" podUID="65b78e2c-855a-49e2-9de8-4fc5a8cfe984" containerName="revoke-keys" Oct 30 19:55:40.647748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.647688 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b78e2c-855a-49e2-9de8-4fc5a8cfe984" containerName="revoke-keys" Oct 30 19:55:40.647869 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.647859 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b78e2c-855a-49e2-9de8-4fc5a8cfe984" containerName="revoke-keys" Oct 30 19:55:40.647904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.647876 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="635573ca-0a7b-4a62-85d2-17ade59afb1f" containerName="istio-proxy" Oct 30 19:55:40.652068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.652048 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.675245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.674956 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 19:55:40.748545 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.748518 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf314f-openshift-default-dockercfg-sjttg\"" Oct 30 19:55:40.789631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.789604 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.789770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.789751 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.789834 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.789800 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.789969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.789951 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.790042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.790008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.790082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.790054 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lxt\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.790189 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.790153 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.790250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.790199 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.790250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.790232 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.836134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.836110 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" Oct 30 19:55:40.836261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.836149 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn" event={"ID":"65b78e2c-855a-49e2-9de8-4fc5a8cfe984","Type":"ContainerDied","Data":"b13a674bd8228956668c505c35a81da4a50f0d429234832773487baafa52ad79"} Oct 30 19:55:40.836261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.836205 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b13a674bd8228956668c505c35a81da4a50f0d429234832773487baafa52ad79" Oct 30 19:55:40.891411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891387 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891435 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891458 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lxt\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891510 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891537 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891570 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891664 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891800 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.891897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891870 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.892085 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.891965 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.892514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.892364 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.892514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.892453 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.892514 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.892471 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.892720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.892628 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.894617 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.894594 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.894737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.894715 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.908339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.908287 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.909053 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.909030 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lxt\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:40.968275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:40.968253 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:41.300379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.300341 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 19:55:41.301065 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:41.301036 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0ac18d_5b71_4bc1_8ec4_5d92ded4f351.slice/crio-18bb52732b7adbee25492f62460e303521e28f4dd6d3d13d44769fe9e2c7d8a6 WatchSource:0}: Error finding container 18bb52732b7adbee25492f62460e303521e28f4dd6d3d13d44769fe9e2c7d8a6: Status 404 returned error can't find the container with id 18bb52732b7adbee25492f62460e303521e28f4dd6d3d13d44769fe9e2c7d8a6 Oct 30 19:55:41.305016 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.304982 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:41.305181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.305049 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:41.305181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.305082 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:55:41.639713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.639448 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:41.847518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.847484 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" event={"ID":"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351","Type":"ContainerStarted","Data":"8d8d15b5934c13c4d37f9a9eca5b9b1e628d802cb2a2a10d884ea4bdcd028232"} Oct 30 19:55:41.847686 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.847538 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" event={"ID":"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351","Type":"ContainerStarted","Data":"18bb52732b7adbee25492f62460e303521e28f4dd6d3d13d44769fe9e2c7d8a6"} Oct 30 19:55:41.889875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.889767 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podStartSLOduration=1.8897498750000001 podStartE2EDuration="1.889749875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:41.883859603 +0000 UTC m=+3049.595384972" watchObservedRunningTime="2026-10-30 19:55:41.889749875 +0000 UTC m=+3049.601275242" Oct 30 19:55:41.969240 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:41.969207 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:42.630932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:42.630893 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" podUID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" containerName="main" containerID="cri-o://cdef5cd5b77789d66705fdaabeff46569dd7db79685c1d67e9a4deb770f051aa" gracePeriod=45 Oct 30 19:55:42.922011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:42.920024 4990 generic.go:356] "Generic (PLEG): container finished" podID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" containerID="cdef5cd5b77789d66705fdaabeff46569dd7db79685c1d67e9a4deb770f051aa" exitCode=0 Oct 30 19:55:42.922011 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:42.920431 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" event={"ID":"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21","Type":"ContainerDied","Data":"cdef5cd5b77789d66705fdaabeff46569dd7db79685c1d67e9a4deb770f051aa"} Oct 30 19:55:42.984128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:42.983208 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:42.984128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:42.983269 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:43.038786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.038753 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:55:43.140891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.140853 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141005 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141143 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141183 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141163 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.141356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141215 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\" (UniqueName: \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141320 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.141470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141406 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141446 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\") pod \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\" (UID: \"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21\") " Oct 30 19:55:43.141879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.141837 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.142266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.142242 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.142357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.142269 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.142357 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.142283 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.144566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.144196 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:43.144566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.144240 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "kube-api-access-n2w6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:43.144885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.144861 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm" pod "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" (UID: "d0856eda-3688-40b3-9e1c-b0bc2ae5ab21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:43.243570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.243485 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.243570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.243518 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2w6k\" (UniqueName: \"kubernetes.io/projected/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-kube-api-access-n2w6k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.243570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.243529 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:43.362052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.362019 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:43.931735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.931707 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" Oct 30 19:55:43.932215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.931747 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn" event={"ID":"d0856eda-3688-40b3-9e1c-b0bc2ae5ab21","Type":"ContainerDied","Data":"ca5952af3d5bc692af3d8060e5fe90d27365bd20019f6d4c0c5caf0c991e8ad0"} Oct 30 19:55:43.932215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.931799 4990 scope.go:122] "RemoveContainer" containerID="cdef5cd5b77789d66705fdaabeff46569dd7db79685c1d67e9a4deb770f051aa" Oct 30 19:55:43.986128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.985843 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:43.986128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:43.985903 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:44.002295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:44.002214 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:55:44.007477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:44.007452 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/sub-test-model-870193-kserve-6599d7b4f-lrcmn"] Oct 30 19:55:44.417031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:44.416833 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" path="/var/lib/kubelet/pods/d0856eda-3688-40b3-9e1c-b0bc2ae5ab21/volumes" Oct 30 19:55:44.973115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:44.972422 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:44.973115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:44.972480 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:45.017516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:45.017434 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.981115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:45.979206 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:45.981115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:45.979269 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:46.654460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:46.654426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:46.984172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:46.979676 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:46.984172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:46.979763 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:47.404542 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.404503 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp"] Oct 30 19:55:47.406182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.406155 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" containerName="main" Oct 30 19:55:47.406352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.406338 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" containerName="main" Oct 30 19:55:47.406697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.406677 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0856eda-3688-40b3-9e1c-b0bc2ae5ab21" containerName="main" Oct 30 19:55:47.413391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.413363 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.416048 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.416014 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp"] Oct 30 19:55:47.437125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.436864 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:55:47.437125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.436865 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:47.450838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.450796 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829p6\" (UniqueName: \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.450838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.450834 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.451032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.451001 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.552576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.552540 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829p6\" (UniqueName: \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.552576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.552581 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.552812 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.552710 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.553732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.553708 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.555214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.555194 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.585923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.585882 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829p6\" (UniqueName: \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\") pod \"maas-api-revoke-sub-effe227d-s9zpp\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.734075 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.733989 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:47.969127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.969065 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Oct 30 19:55:47.969372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:47.969337 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Oct 30 19:55:48.077184 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:48.076634 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a8eda6_66a4_47db_a4b2_c9f2c6e424b7.slice/crio-e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713 WatchSource:0}: Error finding container e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713: Status 404 returned error can't find the container with id e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713 Oct 30 19:55:48.078495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.078461 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp"] Oct 30 19:55:48.873204 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.873113 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.975132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.975054 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:48.975469 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.975438 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:48.976452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.976430 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 19:55:48.994355 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.994314 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" event={"ID":"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7","Type":"ContainerStarted","Data":"dc6d38827ae3ffb383ef21e3cb3dfcbd7593983f880b58a2d5cea62d9348aad8"} Oct 30 19:55:48.994567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:48.994382 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" event={"ID":"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7","Type":"ContainerStarted","Data":"e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713"} Oct 30 19:55:49.112483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:49.112419 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" podStartSLOduration=2.112398558 podStartE2EDuration="2.112398558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:49.109506694 +0000 UTC m=+3056.821032061" watchObservedRunningTime="2026-10-30 19:55:49.112398558 +0000 UTC m=+3056.823923926" Oct 30 19:55:50.005288 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:50.005252 4990 generic.go:356] "Generic (PLEG): container finished" podID="42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" containerID="dc6d38827ae3ffb383ef21e3cb3dfcbd7593983f880b58a2d5cea62d9348aad8" exitCode=0 Oct 30 19:55:50.005508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:50.005336 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" event={"ID":"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7","Type":"ContainerDied","Data":"dc6d38827ae3ffb383ef21e3cb3dfcbd7593983f880b58a2d5cea62d9348aad8"} Oct 30 19:55:50.791767 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:50.791729 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:55:50.822776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:50.822744 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:55:50.982632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:50.982590 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.138281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.138217 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.266752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.266725 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:51.334760 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.334676 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\" (UniqueName: \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\") pod \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " Oct 30 19:55:51.335061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.335034 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\") pod \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " Oct 30 19:55:51.335193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.335173 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\") pod \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\" (UID: \"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7\") " Oct 30 19:55:51.335694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.335597 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca" pod "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" (UID: "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:51.335966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.335947 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:51.337491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.337461 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca" pod "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" (UID: "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:51.337613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.337494 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6" pod "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" (UID: "42a8eda6-66a4-47db-a4b2-c9f2c6e424b7"). InnerVolumeSpecName "kube-api-access-829p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:51.437371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.437337 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829p6\" (UniqueName: \"kubernetes.io/projected/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-kube-api-access-829p6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:51.437371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.437369 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:51.544782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.544634 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.703030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.656336 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2"] Oct 30 19:55:51.703030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.657671 4990 cpu_manager.go:414] "containerMap: removing container" podUID="42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" containerName="revoke-keys" Oct 30 19:55:51.703030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.657696 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" containerName="revoke-keys" Oct 30 19:55:51.703030 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.657970 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" containerName="revoke-keys" Oct 30 19:55:51.705301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.704895 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.718056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.717955 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2"] Oct 30 19:55:51.810206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.810163 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.810630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.810311 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwt4n\" (UniqueName: \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.810630 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.810428 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.912709 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.912666 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwt4n\" (UniqueName: \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.912946 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.912815 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.913156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.913136 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.913751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.913731 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.915305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.915280 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:51.929457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:51.929423 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwt4n\" (UniqueName: \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:52.026830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.026747 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" Oct 30 19:55:52.026830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.026772 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp" event={"ID":"42a8eda6-66a4-47db-a4b2-c9f2c6e424b7","Type":"ContainerDied","Data":"e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713"} Oct 30 19:55:52.026830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.026819 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4752e1e024c41e238c6436920e22eaccebd8287104164f0e3b7f4a52f6fb713" Oct 30 19:55:52.051533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.051510 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:52.362314 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:52.362284 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7f290c_e64c_478a_96a6_ab7599a548d3.slice/crio-cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022 WatchSource:0}: Error finding container cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022: Status 404 returned error can't find the container with id cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022 Oct 30 19:55:52.362477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.362457 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2"] Oct 30 19:55:52.572559 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.572522 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.721628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:52.721547 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:53.038342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:53.038250 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" event={"ID":"3f7f290c-e64c-478a-96a6-ab7599a548d3","Type":"ContainerStarted","Data":"03ea4a1dd558cd18028b962dc31e3dec4990b5cd7654d90fb16b67ddc3951e11"} Oct 30 19:55:53.038342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:53.038336 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" event={"ID":"3f7f290c-e64c-478a-96a6-ab7599a548d3","Type":"ContainerStarted","Data":"cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022"} Oct 30 19:55:53.081524 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:53.081467 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" podStartSLOduration=2.081451355 podStartE2EDuration="2.081451355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:53.078314848 +0000 UTC m=+3060.789840214" watchObservedRunningTime="2026-10-30 19:55:53.081451355 +0000 UTC m=+3060.792976720" Oct 30 19:55:53.576954 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:53.576343 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.049394 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.049352 4990 generic.go:356] "Generic (PLEG): container finished" podID="3f7f290c-e64c-478a-96a6-ab7599a548d3" containerID="03ea4a1dd558cd18028b962dc31e3dec4990b5cd7654d90fb16b67ddc3951e11" exitCode=0 Oct 30 19:55:54.049842 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.049437 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" event={"ID":"3f7f290c-e64c-478a-96a6-ab7599a548d3","Type":"ContainerDied","Data":"03ea4a1dd558cd18028b962dc31e3dec4990b5cd7654d90fb16b67ddc3951e11"} Oct 30 19:55:54.311911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.311817 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.452964 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.452926 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:55:54.457859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.457829 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.482529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.476925 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-315ea2\"" Oct 30 19:55:54.482529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.477132 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-315ea2-dockercfg-l9xnd\"" Oct 30 19:55:54.482529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.481381 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.489594 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.489551 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:55:54.555123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.554554 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.555123 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.554748 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5q6\" (UniqueName: \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.694173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.668920 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5q6\" (UniqueName: \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.712741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.704705 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.712741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.705854 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.732840 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.732546 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5q6\" (UniqueName: \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:54.790228 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:54.789619 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:55.122662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.122626 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:55:55.123961 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:55:55.123931 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2b499c_d5a1_4196_8971_bd5737df4ec7.slice/crio-e82b6474b9c72a616dbae3d2af1c9f1ea9c73768ada3a8feb700af549ed03f3a WatchSource:0}: Error finding container e82b6474b9c72a616dbae3d2af1c9f1ea9c73768ada3a8feb700af549ed03f3a: Status 404 returned error can't find the container with id e82b6474b9c72a616dbae3d2af1c9f1ea9c73768ada3a8feb700af549ed03f3a Oct 30 19:55:55.239651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.239624 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:55.316134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.316013 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\") pod \"3f7f290c-e64c-478a-96a6-ab7599a548d3\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " Oct 30 19:55:55.316314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.316166 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\") pod \"3f7f290c-e64c-478a-96a6-ab7599a548d3\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " Oct 30 19:55:55.316550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.316510 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\" (UniqueName: \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\") pod \"3f7f290c-e64c-478a-96a6-ab7599a548d3\" (UID: \"3f7f290c-e64c-478a-96a6-ab7599a548d3\") " Oct 30 19:55:55.316638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.316586 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca" pod "3f7f290c-e64c-478a-96a6-ab7599a548d3" (UID: "3f7f290c-e64c-478a-96a6-ab7599a548d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:55.317596 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.317564 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:55.318780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.318747 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n" pod "3f7f290c-e64c-478a-96a6-ab7599a548d3" (UID: "3f7f290c-e64c-478a-96a6-ab7599a548d3"). InnerVolumeSpecName "kube-api-access-hwt4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:55.319109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.319064 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca" pod "3f7f290c-e64c-478a-96a6-ab7599a548d3" (UID: "3f7f290c-e64c-478a-96a6-ab7599a548d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:55.419733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.419694 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwt4n\" (UniqueName: \"kubernetes.io/projected/3f7f290c-e64c-478a-96a6-ab7599a548d3-kube-api-access-hwt4n\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:55.419733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.419731 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7f290c-e64c-478a-96a6-ab7599a548d3-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:55.613466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.613369 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:55.792651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.792604 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" podUID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" containerName="payload-processing" containerID="cri-o://189ae794e7232fe6dabe1cd5e70a9878f17244520e9c68f223ac97daf1ff86ca" gracePeriod=25 Oct 30 19:55:55.824045 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:55.823592 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" podUID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" containerName="payload-pre-processing" containerID="cri-o://75d978e34d145e6a2e7b922e5ea4e526d1e0603b1eeef334519f0d70c9ecdf52" gracePeriod=25 Oct 30 19:55:56.098894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.097812 4990 generic.go:356] "Generic (PLEG): container finished" podID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" containerID="189ae794e7232fe6dabe1cd5e70a9878f17244520e9c68f223ac97daf1ff86ca" exitCode=0 Oct 30 19:55:56.098894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.097953 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" event={"ID":"d3b29685-2d71-46fe-a51d-b0f12d864fa4","Type":"ContainerDied","Data":"189ae794e7232fe6dabe1cd5e70a9878f17244520e9c68f223ac97daf1ff86ca"} Oct 30 19:55:56.104182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.100663 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" Oct 30 19:55:56.104182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.100675 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2" event={"ID":"3f7f290c-e64c-478a-96a6-ab7599a548d3","Type":"ContainerDied","Data":"cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022"} Oct 30 19:55:56.104182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.100722 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3afd7a5720f883c8c53365a4c855297ed9a6aaf24fe6b508dd7152c12db022" Oct 30 19:55:56.104182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.103350 4990 generic.go:356] "Generic (PLEG): container finished" podID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" containerID="75d978e34d145e6a2e7b922e5ea4e526d1e0603b1eeef334519f0d70c9ecdf52" exitCode=0 Oct 30 19:55:56.104182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.103571 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" event={"ID":"f8f5b116-f17d-46f4-8d0d-80ddd51e6690","Type":"ContainerDied","Data":"75d978e34d145e6a2e7b922e5ea4e526d1e0603b1eeef334519f0d70c9ecdf52"} Oct 30 19:55:56.106876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.106843 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" event={"ID":"3c2b499c-d5a1-4196-8971-bd5737df4ec7","Type":"ContainerStarted","Data":"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a"} Oct 30 19:55:56.107037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.106909 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:55:56.107037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.106937 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" event={"ID":"3c2b499c-d5a1-4196-8971-bd5737df4ec7","Type":"ContainerStarted","Data":"e82b6474b9c72a616dbae3d2af1c9f1ea9c73768ada3a8feb700af549ed03f3a"} Oct 30 19:55:56.160460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.154019 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" podStartSLOduration=2.153965151 podStartE2EDuration="2.153965151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:56.14587063 +0000 UTC m=+3063.857395996" watchObservedRunningTime="2026-10-30 19:55:56.153965151 +0000 UTC m=+3063.865490518" Oct 30 19:55:56.187156 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.187109 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:55:56.209345 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.209303 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:55:56.209613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.209425 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:55:56.241959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.241921 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\" (UniqueName: \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\") pod \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " Oct 30 19:55:56.242556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.242519 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\") pod \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\" (UID: \"d3b29685-2d71-46fe-a51d-b0f12d864fa4\") " Oct 30 19:55:56.242961 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.242934 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume" pod "d3b29685-2d71-46fe-a51d-b0f12d864fa4" (UID: "d3b29685-2d71-46fe-a51d-b0f12d864fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:56.243573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.243544 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b29685-2d71-46fe-a51d-b0f12d864fa4-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:56.244461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.244435 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf" pod "d3b29685-2d71-46fe-a51d-b0f12d864fa4" (UID: "d3b29685-2d71-46fe-a51d-b0f12d864fa4"). InnerVolumeSpecName "kube-api-access-5mcrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:56.245894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.245864 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.421766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.421635 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\" (UniqueName: \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\") pod \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " Oct 30 19:55:56.422174 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.422145 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\") pod \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\" (UID: \"f8f5b116-f17d-46f4-8d0d-80ddd51e6690\") " Oct 30 19:55:56.423462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.423370 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcrf\" (UniqueName: \"kubernetes.io/projected/d3b29685-2d71-46fe-a51d-b0f12d864fa4-kube-api-access-5mcrf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:56.424103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.424004 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume" pod "f8f5b116-f17d-46f4-8d0d-80ddd51e6690" (UID: "f8f5b116-f17d-46f4-8d0d-80ddd51e6690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:56.432173 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.427815 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst" pod "f8f5b116-f17d-46f4-8d0d-80ddd51e6690" (UID: "f8f5b116-f17d-46f4-8d0d-80ddd51e6690"). InnerVolumeSpecName "kube-api-access-5pwst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:56.530199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.527644 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:56.530199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:56.527688 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwst\" (UniqueName: \"kubernetes.io/projected/f8f5b116-f17d-46f4-8d0d-80ddd51e6690-kube-api-access-5pwst\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:55:57.120984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.120949 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" Oct 30 19:55:57.121220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.120984 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv" event={"ID":"f8f5b116-f17d-46f4-8d0d-80ddd51e6690","Type":"ContainerDied","Data":"350102254a2e4fb53e2cc7ee558c8d4e05aebbb83c31095898d36f3da8a19ceb"} Oct 30 19:55:57.121220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.121070 4990 scope.go:122] "RemoveContainer" containerID="75d978e34d145e6a2e7b922e5ea4e526d1e0603b1eeef334519f0d70c9ecdf52" Oct 30 19:55:57.124420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.123731 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" Oct 30 19:55:57.124420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.124176 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp" event={"ID":"d3b29685-2d71-46fe-a51d-b0f12d864fa4","Type":"ContainerDied","Data":"32c834593d5f726178446979face21ee248f1d0321cdd285eaa74766cd4765f3"} Oct 30 19:55:57.137910 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.137875 4990 scope.go:122] "RemoveContainer" containerID="189ae794e7232fe6dabe1cd5e70a9878f17244520e9c68f223ac97daf1ff86ca" Oct 30 19:55:57.168702 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.168665 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:55:57.178561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.178523 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-fbf44d548-grdrp"] Oct 30 19:55:57.208452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.208413 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:55:57.217311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.217272 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-65744d55d7-4n2rv"] Oct 30 19:55:57.348112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:57.345978 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:58.403665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:58.403629 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" path="/var/lib/kubelet/pods/d3b29685-2d71-46fe-a51d-b0f12d864fa4/volumes" Oct 30 19:55:58.404160 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:58.404144 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" path="/var/lib/kubelet/pods/f8f5b116-f17d-46f4-8d0d-80ddd51e6690/volumes" Oct 30 19:55:58.928932 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:55:58.928893 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:00.550190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:00.550150 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.033427 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:01.033335 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85"] Oct 30 19:56:01.039803 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:01.039771 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5a237d-q2m85"] Oct 30 19:56:02.356297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:02.356262 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:02.405497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:02.405461 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29950bf1-1294-47ba-908f-3c594a58b572" path="/var/lib/kubelet/pods/29950bf1-1294-47ba-908f-3c594a58b572/volumes" Oct 30 19:56:02.493416 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:02.493375 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:02.631437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:02.631342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.070338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:04.069621 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.100517 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:05.100461 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:56:05.623386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:05.623342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.128063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:07.128031 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:56:07.321220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:07.321172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.951642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:08.951603 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:09.187549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:09.187499 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:56:09.261797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:09.261718 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.729851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:10.729808 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:11.234603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.234563 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" podUID="75985826-9fc3-474e-b98c-9d3874e0ad80" containerName="main" containerID="cri-o://17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a" gracePeriod=45 Oct 30 19:56:11.525700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.525673 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:56:11.623707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.623666 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.623902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.623769 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.623902 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.623856 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\" (UniqueName: \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.624037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.623901 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.624037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.623923 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:11.624190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624061 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:11.624190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624124 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\" (UniqueName: \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.624190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624167 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\") pod \"75985826-9fc3-474e-b98c-9d3874e0ad80\" (UID: \"75985826-9fc3-474e-b98c-9d3874e0ad80\") " Oct 30 19:56:11.624583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624555 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:11.624911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624888 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:11.625014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624915 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:11.625014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.624930 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:11.626488 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.626462 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:11.626666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.626644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "kube-api-access-n5dpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:11.626916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.626894 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm" pod "75985826-9fc3-474e-b98c-9d3874e0ad80" (UID: "75985826-9fc3-474e-b98c-9d3874e0ad80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:11.726001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.725964 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75985826-9fc3-474e-b98c-9d3874e0ad80-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:11.726001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.725996 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75985826-9fc3-474e-b98c-9d3874e0ad80-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:11.726001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:11.726009 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dpg\" (UniqueName: \"kubernetes.io/projected/75985826-9fc3-474e-b98c-9d3874e0ad80-kube-api-access-n5dpg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:12.031917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.031874 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb"] Oct 30 19:56:12.043130 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.043083 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0b648a-6cptb"] Oct 30 19:56:12.277831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.277787 4990 generic.go:356] "Generic (PLEG): container finished" podID="75985826-9fc3-474e-b98c-9d3874e0ad80" containerID="17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a" exitCode=0 Oct 30 19:56:12.278052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.277859 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" Oct 30 19:56:12.278052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.277879 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" event={"ID":"75985826-9fc3-474e-b98c-9d3874e0ad80","Type":"ContainerDied","Data":"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a"} Oct 30 19:56:12.278052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.277925 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r" event={"ID":"75985826-9fc3-474e-b98c-9d3874e0ad80","Type":"ContainerDied","Data":"cac3a0068e8d80cd713d762b1c51076264114ca740da4c316e36a152c97396f0"} Oct 30 19:56:12.278052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.277943 4990 scope.go:122] "RemoveContainer" containerID="17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a" Oct 30 19:56:12.293587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.293562 4990 scope.go:122] "RemoveContainer" containerID="17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a" Oct 30 19:56:12.293971 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:12.293941 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a\": container with ID starting with 17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a not found: ID does not exist" containerID="17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a" Oct 30 19:56:12.294058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.294031 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a"} err="failed to get container status \"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a\": rpc error: code = NotFound desc = could not find container \"17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a\": container with ID starting with 17af1194d0a471f9a054c776afa6af2baecd6f68f45e1698bb78be584751291a not found: ID does not exist" Oct 30 19:56:12.325324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.325290 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:56:12.335072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.335045 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-ec2fe7/auth-test-model-ec2fe7-kserve-64c784bb69-29m2r"] Oct 30 19:56:12.406967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.406927 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df6810e-f549-4d9e-8cb0-b1abeb8847cc" path="/var/lib/kubelet/pods/3df6810e-f549-4d9e-8cb0-b1abeb8847cc/volumes" Oct 30 19:56:12.407592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:12.407568 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75985826-9fc3-474e-b98c-9d3874e0ad80" path="/var/lib/kubelet/pods/75985826-9fc3-474e-b98c-9d3874e0ad80/volumes" Oct 30 19:56:13.078373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:13.078337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.063822 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:14.063785 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:56:14.100276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:14.100241 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:56:14.256145 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:14.256109 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.410261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:14.410177 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.834009 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:14.833974 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.517430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:16.515948 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.050184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:18.050147 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.064949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.064903 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" podUID="6d33c656-b26e-429c-bbc2-2f76974ffd11" containerName="payload-processing" containerID="cri-o://809bcfc6ce09e1587ff59f00f5eb4b9cad0222e152b24de451fe9dcf6b11f693" gracePeriod=25 Oct 30 19:56:19.100776 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.100692 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" podUID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" containerName="payload-pre-processing" containerID="cri-o://8e831a913bc685ba3b2705c0b65234aca8244f854829fd2e3a187cda3228f2ca" gracePeriod=25 Oct 30 19:56:19.353832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.353793 4990 generic.go:356] "Generic (PLEG): container finished" podID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" containerID="8e831a913bc685ba3b2705c0b65234aca8244f854829fd2e3a187cda3228f2ca" exitCode=0 Oct 30 19:56:19.354363 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.354331 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" event={"ID":"6135aae8-d2e3-41b6-860b-50a7c13fbd7d","Type":"ContainerDied","Data":"8e831a913bc685ba3b2705c0b65234aca8244f854829fd2e3a187cda3228f2ca"} Oct 30 19:56:19.356602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.356577 4990 generic.go:356] "Generic (PLEG): container finished" podID="6d33c656-b26e-429c-bbc2-2f76974ffd11" containerID="809bcfc6ce09e1587ff59f00f5eb4b9cad0222e152b24de451fe9dcf6b11f693" exitCode=0 Oct 30 19:56:19.356748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.356651 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" event={"ID":"6d33c656-b26e-429c-bbc2-2f76974ffd11","Type":"ContainerDied","Data":"809bcfc6ce09e1587ff59f00f5eb4b9cad0222e152b24de451fe9dcf6b11f693"} Oct 30 19:56:19.619007 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.618843 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:56:19.634148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.634086 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:56:19.675510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.675472 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\" (UniqueName: \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\") pod \"6d33c656-b26e-429c-bbc2-2f76974ffd11\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " Oct 30 19:56:19.675729 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.675702 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\") pod \"6d33c656-b26e-429c-bbc2-2f76974ffd11\" (UID: \"6d33c656-b26e-429c-bbc2-2f76974ffd11\") " Oct 30 19:56:19.676196 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.675951 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\") pod \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " Oct 30 19:56:19.676598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.676078 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume" pod "6d33c656-b26e-429c-bbc2-2f76974ffd11" (UID: "6d33c656-b26e-429c-bbc2-2f76974ffd11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:19.676734 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.676479 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume" pod "6135aae8-d2e3-41b6-860b-50a7c13fbd7d" (UID: "6135aae8-d2e3-41b6-860b-50a7c13fbd7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:19.677059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.677037 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\" (UniqueName: \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\") pod \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\" (UID: \"6135aae8-d2e3-41b6-860b-50a7c13fbd7d\") " Oct 30 19:56:19.678446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.678423 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d33c656-b26e-429c-bbc2-2f76974ffd11-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:19.678748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.678727 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:19.678748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.678685 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx" pod "6d33c656-b26e-429c-bbc2-2f76974ffd11" (UID: "6d33c656-b26e-429c-bbc2-2f76974ffd11"). InnerVolumeSpecName "kube-api-access-zrfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:19.679665 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.679639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr" pod "6135aae8-d2e3-41b6-860b-50a7c13fbd7d" (UID: "6135aae8-d2e3-41b6-860b-50a7c13fbd7d"). InnerVolumeSpecName "kube-api-access-zmcrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:19.780014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.779974 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcrr\" (UniqueName: \"kubernetes.io/projected/6135aae8-d2e3-41b6-860b-50a7c13fbd7d-kube-api-access-zmcrr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:19.780014 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.780009 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfqx\" (UniqueName: \"kubernetes.io/projected/6d33c656-b26e-429c-bbc2-2f76974ffd11-kube-api-access-zrfqx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:19.851920 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:19.851879 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.326798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.326755 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.470372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.470276 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" Oct 30 19:56:20.473568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.473538 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" Oct 30 19:56:20.475949 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.475797 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z" event={"ID":"6d33c656-b26e-429c-bbc2-2f76974ffd11","Type":"ContainerDied","Data":"ffb6c44986446845a39cdc805f23e19e9ea0a1583299087c64dd13042af978c6"} Oct 30 19:56:20.476184 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.476162 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69" event={"ID":"6135aae8-d2e3-41b6-860b-50a7c13fbd7d","Type":"ContainerDied","Data":"4ee4c3cc505a9f36845350442d677fe86d2407f62cc12f9278d29977fddb9152"} Oct 30 19:56:20.476323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.476310 4990 scope.go:122] "RemoveContainer" containerID="809bcfc6ce09e1587ff59f00f5eb4b9cad0222e152b24de451fe9dcf6b11f693" Oct 30 19:56:20.493862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.493837 4990 scope.go:122] "RemoveContainer" containerID="8e831a913bc685ba3b2705c0b65234aca8244f854829fd2e3a187cda3228f2ca" Oct 30 19:56:20.530931 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.530894 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:56:20.540797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.540764 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7857797974-whl9z"] Oct 30 19:56:20.577226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.577192 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:56:20.584312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.584283 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-96d7db567-q4x69"] Oct 30 19:56:20.635376 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.635328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.728169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.727535 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:56:20.814704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.814658 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.826646 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.826456 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx"] Oct 30 19:56:20.828388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828364 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3f7f290c-e64c-478a-96a6-ab7599a548d3" containerName="revoke-keys" Oct 30 19:56:20.828554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828523 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7f290c-e64c-478a-96a6-ab7599a548d3" containerName="revoke-keys" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828578 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" containerName="payload-pre-processing" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828589 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" containerName="payload-pre-processing" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828616 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" containerName="payload-pre-processing" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828624 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" containerName="payload-pre-processing" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828649 4990 cpu_manager.go:414] "containerMap: removing container" podUID="75985826-9fc3-474e-b98c-9d3874e0ad80" containerName="main" Oct 30 19:56:20.828705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828657 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75985826-9fc3-474e-b98c-9d3874e0ad80" containerName="main" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828747 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6d33c656-b26e-429c-bbc2-2f76974ffd11" containerName="payload-processing" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828759 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d33c656-b26e-429c-bbc2-2f76974ffd11" containerName="payload-processing" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828829 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" containerName="payload-processing" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.828837 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" containerName="payload-processing" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829285 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b29685-2d71-46fe-a51d-b0f12d864fa4" containerName="payload-processing" Oct 30 19:56:20.829322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829322 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7f290c-e64c-478a-96a6-ab7599a548d3" containerName="revoke-keys" Oct 30 19:56:20.829905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829342 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" containerName="payload-pre-processing" Oct 30 19:56:20.829905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829354 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f5b116-f17d-46f4-8d0d-80ddd51e6690" containerName="payload-pre-processing" Oct 30 19:56:20.829905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829372 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="75985826-9fc3-474e-b98c-9d3874e0ad80" containerName="main" Oct 30 19:56:20.829905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.829407 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d33c656-b26e-429c-bbc2-2f76974ffd11" containerName="payload-processing" Oct 30 19:56:20.957182 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.957145 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx"] Oct 30 19:56:20.957502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.957485 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:20.970894 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.970838 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:20.971311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:20.971289 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:56:21.032274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.032190 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.032464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.032272 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.032464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.032322 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjtc\" (UniqueName: \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.134150 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.134105 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.134371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.134230 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.134371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.134339 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjtc\" (UniqueName: \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.134995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.134964 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.137546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.137517 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.151975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.151942 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjtc\" (UniqueName: \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.305205 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.305160 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:21.533285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.459318 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.998236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:21.998180 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 19:56:22.044115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.041383 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.098263 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.098216 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx"] Oct 30 19:56:22.098666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.098636 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.119892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.119300 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 19:56:22.148257 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.148166 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fe3e5e-openshift-default-dockercfg-pdlsc\"" Oct 30 19:56:22.172382 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172340 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172476 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172555 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172586 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172730 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172765 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172804 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172842 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpz77\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.172927 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.172912 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.187447 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.187409 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.274976 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.274888 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275238 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275309 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275366 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpz77\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275495 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275639 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275726 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275815 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275848 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.275728 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.276032 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.276325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.276325 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.277060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.276713 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.277060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.277044 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.368848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.368772 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.370473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.370416 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.372186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.372154 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpz77\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.376741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.376715 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\") pod \"e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.419010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.418974 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6135aae8-d2e3-41b6-860b-50a7c13fbd7d" path="/var/lib/kubelet/pods/6135aae8-d2e3-41b6-860b-50a7c13fbd7d/volumes" Oct 30 19:56:22.420038 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.420011 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d33c656-b26e-429c-bbc2-2f76974ffd11" path="/var/lib/kubelet/pods/6d33c656-b26e-429c-bbc2-2f76974ffd11/volumes" Oct 30 19:56:22.435347 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.435321 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:22.447477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:22.447435 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.090701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.090654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" event={"ID":"21ec35e5-62a3-4ee1-b73f-98b264bec799","Type":"ContainerStarted","Data":"d3d75fb24ec5ad253a801022de04b3ff6751d5e40e1b1d35fd193f2488752880"} Oct 30 19:56:23.091351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.091321 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" event={"ID":"21ec35e5-62a3-4ee1-b73f-98b264bec799","Type":"ContainerStarted","Data":"7760e31b0d5d0f7d392f81d58518b82cd4014471a57b1310518ababa5627b340"} Oct 30 19:56:23.242770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.242703 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" podStartSLOduration=3.242683895 podStartE2EDuration="3.242683895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:23.225784974 +0000 UTC m=+3090.937310340" watchObservedRunningTime="2026-10-30 19:56:23.242683895 +0000 UTC m=+3090.954209262" Oct 30 19:56:23.320244 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:56:23.320204 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a85259_5ff8_4caf_8d5b_195e5ead1504.slice/crio-4785806944c01b16357bf7a72054d26aff9df21db2c5fa3657e129c3467fb352 WatchSource:0}: Error finding container 4785806944c01b16357bf7a72054d26aff9df21db2c5fa3657e129c3467fb352: Status 404 returned error can't find the container with id 4785806944c01b16357bf7a72054d26aff9df21db2c5fa3657e129c3467fb352 Oct 30 19:56:23.327342 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.327307 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 19:56:23.328461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.328416 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:56:23.328587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.328508 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:56:23.328587 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.328552 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:56:23.789417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:23.789295 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.252115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.250500 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerName="main" containerID="cri-o://256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47" gracePeriod=45 Oct 30 19:56:24.262176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.262121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerStarted","Data":"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b"} Oct 30 19:56:24.269051 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.269014 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerStarted","Data":"4785806944c01b16357bf7a72054d26aff9df21db2c5fa3657e129c3467fb352"} Oct 30 19:56:24.270562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.270532 4990 generic.go:356] "Generic (PLEG): container finished" podID="21ec35e5-62a3-4ee1-b73f-98b264bec799" containerID="d3d75fb24ec5ad253a801022de04b3ff6751d5e40e1b1d35fd193f2488752880" exitCode=0 Oct 30 19:56:24.270750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.270609 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" event={"ID":"21ec35e5-62a3-4ee1-b73f-98b264bec799","Type":"ContainerDied","Data":"d3d75fb24ec5ad253a801022de04b3ff6751d5e40e1b1d35fd193f2488752880"} Oct 30 19:56:24.358754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.358693 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podStartSLOduration=3.358676496 podStartE2EDuration="3.358676496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:24.322953772 +0000 UTC m=+3092.034479139" watchObservedRunningTime="2026-10-30 19:56:24.358676496 +0000 UTC m=+3092.070201861" Oct 30 19:56:24.373703 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.372945 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.248:8000/ready\": dial tcp 10.134.0.248:8000: connect: connection refused" Oct 30 19:56:24.394685 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.394601 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.436664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.436616 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:24.460066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.460026 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:24.544921 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.544801 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.560232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.550753 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2"] Oct 30 19:56:24.560232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.559300 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f44f74-zjcf2"] Oct 30 19:56:24.630190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.630154 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:56:24.688084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688032 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\" (UniqueName: \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688255 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688292 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688333 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688405 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688457 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\") pod \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\" (UID: \"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5\") " Oct 30 19:56:24.688738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.688690 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:24.689141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.689034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:24.689262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.689174 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:24.689262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.689226 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.689262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.689246 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.691421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.691383 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:24.691650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.691624 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "kube-api-access-pg58m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:24.693453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.693424 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs" pod "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" (UID: "0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:24.746657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.746544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.790766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.790728 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.790766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.790762 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.790766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.790772 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.791028 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:24.790781 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg58m\" (UniqueName: \"kubernetes.io/projected/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5-kube-api-access-pg58m\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:25.090364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.090323 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:25.283593 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.283559 4990 generic.go:356] "Generic (PLEG): container finished" podID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerID="256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47" exitCode=0 Oct 30 19:56:25.284072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.283605 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" Oct 30 19:56:25.284072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.283649 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" event={"ID":"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5","Type":"ContainerDied","Data":"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47"} Oct 30 19:56:25.284072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.283708 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w" event={"ID":"0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5","Type":"ContainerDied","Data":"085fa65ce9a1a271bc75fa3cc613591ae387bd096b14cd36a87a5a1f6e6c6c5c"} Oct 30 19:56:25.284072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.283732 4990 scope.go:122] "RemoveContainer" containerID="256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47" Oct 30 19:56:25.284511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.284492 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:25.285616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.285589 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:56:25.299291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.299267 4990 scope.go:122] "RemoveContainer" containerID="256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47" Oct 30 19:56:25.299615 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:25.299596 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47\": container with ID starting with 256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47 not found: ID does not exist" containerID="256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47" Oct 30 19:56:25.299693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.299627 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47"} err="failed to get container status \"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47\": rpc error: code = NotFound desc = could not find container \"256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47\": container with ID starting with 256c1804b8b0598ead9d91acc46bfed14b45a0edd0ba760e4929805a462e0b47 not found: ID does not exist" Oct 30 19:56:25.384952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.384912 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:56:25.391795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.391757 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-5250ee/auth-test-model-5250ee-kserve-67c657f985-4597w"] Oct 30 19:56:25.478195 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.478161 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:25.602789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.602696 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\") pod \"21ec35e5-62a3-4ee1-b73f-98b264bec799\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " Oct 30 19:56:25.602978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.602823 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\") pod \"21ec35e5-62a3-4ee1-b73f-98b264bec799\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " Oct 30 19:56:25.603071 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.603040 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\" (UniqueName: \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\") pod \"21ec35e5-62a3-4ee1-b73f-98b264bec799\" (UID: \"21ec35e5-62a3-4ee1-b73f-98b264bec799\") " Oct 30 19:56:25.603176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.603149 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca" pod "21ec35e5-62a3-4ee1-b73f-98b264bec799" (UID: "21ec35e5-62a3-4ee1-b73f-98b264bec799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:25.603762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.603741 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:25.605300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.605274 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc" pod "21ec35e5-62a3-4ee1-b73f-98b264bec799" (UID: "21ec35e5-62a3-4ee1-b73f-98b264bec799"). InnerVolumeSpecName "kube-api-access-pjjtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:25.605413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.605276 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca" pod "21ec35e5-62a3-4ee1-b73f-98b264bec799" (UID: "21ec35e5-62a3-4ee1-b73f-98b264bec799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:25.705693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.705657 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ec35e5-62a3-4ee1-b73f-98b264bec799-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:25.705693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:25.705686 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjtc\" (UniqueName: \"kubernetes.io/projected/21ec35e5-62a3-4ee1-b73f-98b264bec799-kube-api-access-pjjtc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:26.297354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.297324 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" Oct 30 19:56:26.297768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.297360 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx" event={"ID":"21ec35e5-62a3-4ee1-b73f-98b264bec799","Type":"ContainerDied","Data":"7760e31b0d5d0f7d392f81d58518b82cd4014471a57b1310518ababa5627b340"} Oct 30 19:56:26.297768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.297409 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7760e31b0d5d0f7d392f81d58518b82cd4014471a57b1310518ababa5627b340" Oct 30 19:56:26.405765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.405722 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" path="/var/lib/kubelet/pods/0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5/volumes" Oct 30 19:56:26.406391 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.406366 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7f290c-e64c-478a-96a6-ab7599a548d3" path="/var/lib/kubelet/pods/3f7f290c-e64c-478a-96a6-ab7599a548d3/volumes" Oct 30 19:56:26.631135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:26.623113 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:28.489657 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:28.489613 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:30.254727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:30.254693 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:31.290610 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:31.290571 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:31.457818 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:31.457780 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:31.603959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:31.603917 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:31.746129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:31.746074 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.148252 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.148214 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.297115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.294140 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.349651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.349497 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm"] Oct 30 19:56:32.350550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.350523 4990 cpu_manager.go:414] "containerMap: removing container" podUID="21ec35e5-62a3-4ee1-b73f-98b264bec799" containerName="revoke-keys" Oct 30 19:56:32.350738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.350723 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ec35e5-62a3-4ee1-b73f-98b264bec799" containerName="revoke-keys" Oct 30 19:56:32.350871 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.350860 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerName="main" Oct 30 19:56:32.350959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.350948 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerName="main" Oct 30 19:56:32.351282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.351263 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eecc7cd-7a8c-46bf-acf3-7014cafbb2e5" containerName="main" Oct 30 19:56:32.351428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.351417 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ec35e5-62a3-4ee1-b73f-98b264bec799" containerName="revoke-keys" Oct 30 19:56:32.358942 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.358911 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.379619 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.379586 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm"] Oct 30 19:56:32.379924 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.379904 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:56:32.379990 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.379948 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:32.479056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.478941 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjqn\" (UniqueName: \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.479056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.479023 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.479565 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.479530 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.580813 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.580767 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.581042 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.580984 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjqn\" (UniqueName: \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.581147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.581053 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.581701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.581641 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.583915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.583887 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.600140 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.600083 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjqn\" (UniqueName: \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\") pod \"maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:32.690414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:32.690376 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:33.001591 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:33.001518 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm"] Oct 30 19:56:33.002846 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:56:33.002818 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfd7eb6_16e0_4988_a6ca_a89811c3b0b7.slice/crio-f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d WatchSource:0}: Error finding container f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d: Status 404 returned error can't find the container with id f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d Oct 30 19:56:33.288176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:33.280849 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.380155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:33.380121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" event={"ID":"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7","Type":"ContainerStarted","Data":"f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d"} Oct 30 19:56:33.434880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:33.434830 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.392941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:34.392824 4990 generic.go:356] "Generic (PLEG): container finished" podID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" containerID="67e649be8a063f652e17ecaec9149bc4eec86257f10c38d8befceb3be5077432" exitCode=0 Oct 30 19:56:34.392941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:34.392920 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" event={"ID":"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7","Type":"ContainerDied","Data":"67e649be8a063f652e17ecaec9149bc4eec86257f10c38d8befceb3be5077432"} Oct 30 19:56:34.613928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:34.613886 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:35.101296 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:35.101249 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:56:35.601321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.601293 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:35.638585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.638318 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\") pod \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " Oct 30 19:56:35.638585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.638494 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\") pod \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " Oct 30 19:56:35.638830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.638622 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\" (UniqueName: \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\") pod \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\" (UID: \"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7\") " Oct 30 19:56:35.638913 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.638891 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca" pod "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" (UID: "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:35.639498 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.639466 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.644576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.644540 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca" pod "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" (UID: "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:35.646297 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.646263 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn" pod "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" (UID: "8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7"). InnerVolumeSpecName "kube-api-access-kdjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:35.741036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.740924 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.741036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.740970 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjqn\" (UniqueName: \"kubernetes.io/projected/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7-kube-api-access-kdjqn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.757066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.757023 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" containerName="main" containerID="cri-o://e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018" gracePeriod=45 Oct 30 19:56:35.797010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.796965 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.251:8000/ready\": dial tcp 10.134.0.251:8000: connect: connection refused" Oct 30 19:56:35.891401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:35.891351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.075113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.066086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.204489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.204457 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:56:36.285047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285003 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.285287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285057 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.285287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285116 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.285287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285230 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.285497 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285420 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.285560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285499 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\" (UniqueName: \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\") pod \"b2677cbc-5b99-472a-8f84-d36800493e34\" (UID: \"b2677cbc-5b99-472a-8f84-d36800493e34\") " Oct 30 19:56:36.286317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285652 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.286317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285888 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.286317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.285902 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.286815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.286784 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.286908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.286820 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.286908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.286835 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.289027 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.288981 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.289616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.289574 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:36.289786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.289764 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq" pod "b2677cbc-5b99-472a-8f84-d36800493e34" (UID: "b2677cbc-5b99-472a-8f84-d36800493e34"). InnerVolumeSpecName "kube-api-access-zl8fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:36.402165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.401488 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl8fq\" (UniqueName: \"kubernetes.io/projected/b2677cbc-5b99-472a-8f84-d36800493e34-kube-api-access-zl8fq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.402165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.401530 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2677cbc-5b99-472a-8f84-d36800493e34-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.402165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.401543 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2677cbc-5b99-472a-8f84-d36800493e34-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.575636 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.575472 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:56:36.581380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.580824 4990 generic.go:356] "Generic (PLEG): container finished" podID="b2677cbc-5b99-472a-8f84-d36800493e34" containerID="e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018" exitCode=0 Oct 30 19:56:36.581380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.580941 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" Oct 30 19:56:36.594155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.591080 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" event={"ID":"8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7","Type":"ContainerDied","Data":"f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d"} Oct 30 19:56:36.594155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.591163 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9f898f382d127434b0b270bd20fb34dccf599757fe974e20290aa4a982e993d" Oct 30 19:56:36.594155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.591179 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" event={"ID":"b2677cbc-5b99-472a-8f84-d36800493e34","Type":"ContainerDied","Data":"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018"} Oct 30 19:56:36.594155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.591200 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22" event={"ID":"b2677cbc-5b99-472a-8f84-d36800493e34","Type":"ContainerDied","Data":"a9c37ef4a611ea42ad08b942d17831c4db857ec5e55d9c02574dd481f8de3650"} Oct 30 19:56:36.594155 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.591225 4990 scope.go:122] "RemoveContainer" containerID="e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.595283 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596468 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" containerName="main" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596492 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" containerName="main" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596538 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" containerName="revoke-keys" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596543 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" containerName="revoke-keys" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596685 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" containerName="revoke-keys" Oct 30 19:56:36.598134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.596701 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" containerName="main" Oct 30 19:56:36.606740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.606693 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.615568 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.615533 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 19:56:36.619669 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.619640 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf314f\"" Oct 30 19:56:36.619885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.619771 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf314f-dockercfg-tb77v\"" Oct 30 19:56:36.628518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.628472 4990 scope.go:122] "RemoveContainer" containerID="e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018" Oct 30 19:56:36.628964 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:36.628934 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018\": container with ID starting with e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018 not found: ID does not exist" containerID="e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018" Oct 30 19:56:36.629101 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.628977 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018"} err="failed to get container status \"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018\": rpc error: code = NotFound desc = could not find container \"e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018\": container with ID starting with e9f16a55f8b3c8b31918d98262a59c5da6c880eec859c47432eaf603a753a018 not found: ID does not exist" Oct 30 19:56:36.697339 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.697252 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:56:36.703483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.703442 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.703673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.703503 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-5a0a17/auth-test-model-5a0a17-kserve-6b6c8df44f-srr22"] Oct 30 19:56:36.712197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.712151 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2bk\" (UniqueName: \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.712424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.712402 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.813802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.813766 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2bk\" (UniqueName: \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.813979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.813884 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.814836 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.814815 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.831107 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.831062 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2bk\" (UniqueName: \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\") pod \"payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:36.851786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.851745 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.942484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:36.942449 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:37.258281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.258188 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.265177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.265148 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 19:56:37.266693 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:56:37.266666 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff27e004_cb42_4336_81cb_9881e663796b.slice/crio-01a31f041ff40c4fcfbf91a3665cc1b65ad4c2d5ef482c7b7797b082ab286d5b WatchSource:0}: Error finding container 01a31f041ff40c4fcfbf91a3665cc1b65ad4c2d5ef482c7b7797b082ab286d5b: Status 404 returned error can't find the container with id 01a31f041ff40c4fcfbf91a3665cc1b65ad4c2d5ef482c7b7797b082ab286d5b Oct 30 19:56:37.506425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.506386 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.597752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.597712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" event={"ID":"ff27e004-cb42-4336-81cb-9881e663796b","Type":"ContainerStarted","Data":"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea"} Oct 30 19:56:37.597967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.597778 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" event={"ID":"ff27e004-cb42-4336-81cb-9881e663796b","Type":"ContainerStarted","Data":"01a31f041ff40c4fcfbf91a3665cc1b65ad4c2d5ef482c7b7797b082ab286d5b"} Oct 30 19:56:37.597967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.597825 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:37.631494 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:37.631438 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" podStartSLOduration=1.631423239 podStartE2EDuration="1.631423239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:37.630846498 +0000 UTC m=+3105.342371863" watchObservedRunningTime="2026-10-30 19:56:37.631423239 +0000 UTC m=+3105.342948623" Oct 30 19:56:38.404748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.404713 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2677cbc-5b99-472a-8f84-d36800493e34" path="/var/lib/kubelet/pods/b2677cbc-5b99-472a-8f84-d36800493e34/volumes" Oct 30 19:56:38.524269 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.524231 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.660911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.660814 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:56:38.664334 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.664302 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.666940 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.666912 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:38.691152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.691120 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:56:38.693461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.693435 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fe3e5e\"" Oct 30 19:56:38.694317 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.694298 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fe3e5e-dockercfg-m7tjq\"" Oct 30 19:56:38.897129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.887177 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:56:38.988510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.950823 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:38.988510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:38.950898 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb5z\" (UniqueName: \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.015112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.006985 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:56:39.015112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.007125 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.097531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.097487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cms\" (UniqueName: \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.097725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.097653 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.097790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.097722 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.097875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.097819 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb5z\" (UniqueName: \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.098527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.098505 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.115221 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.115185 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb5z\" (UniqueName: \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.199426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.199389 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cms\" (UniqueName: \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.199653 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.199525 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.200336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.200315 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.216878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.216841 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cms\" (UniqueName: \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\") pod \"payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.314037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.313992 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:39.332792 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.332767 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:39.639270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.639237 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:56:39.639423 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:56:39.639277 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c84065_e853_41c1_b85b_f442561dee8e.slice/crio-e0313796c331019bb99ea1dbdd59ea1192cf5adf8f970dece16d5a8a2a482daa WatchSource:0}: Error finding container e0313796c331019bb99ea1dbdd59ea1192cf5adf8f970dece16d5a8a2a482daa: Status 404 returned error can't find the container with id e0313796c331019bb99ea1dbdd59ea1192cf5adf8f970dece16d5a8a2a482daa Oct 30 19:56:39.683779 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.680769 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:56:39.771508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.771269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:39.917510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:39.917409 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:40.630576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.630527 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" event={"ID":"f31826d8-c296-4c75-a2a4-99a1f1c93f95","Type":"ContainerStarted","Data":"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0"} Oct 30 19:56:40.630833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.630606 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" event={"ID":"f31826d8-c296-4c75-a2a4-99a1f1c93f95","Type":"ContainerStarted","Data":"91d69fd3edba2db0f3274d6ef9bd09b8b58677cc12876ee1cdd3a306af5536d8"} Oct 30 19:56:40.630833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.630634 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:40.632409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.632381 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" event={"ID":"d2c84065-e853-41c1-b85b-f442561dee8e","Type":"ContainerStarted","Data":"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69"} Oct 30 19:56:40.632529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.632432 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" event={"ID":"d2c84065-e853-41c1-b85b-f442561dee8e","Type":"ContainerStarted","Data":"e0313796c331019bb99ea1dbdd59ea1192cf5adf8f970dece16d5a8a2a482daa"} Oct 30 19:56:40.632529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.632463 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:40.671832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.671769 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" podStartSLOduration=2.671753798 podStartE2EDuration="2.671753798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:40.665990911 +0000 UTC m=+3108.377516303" watchObservedRunningTime="2026-10-30 19:56:40.671753798 +0000 UTC m=+3108.383279163" Oct 30 19:56:40.707385 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:40.707310 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" podStartSLOduration=2.707291561 podStartE2EDuration="2.707291561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:40.699736656 +0000 UTC m=+3108.411262021" watchObservedRunningTime="2026-10-30 19:56:40.707291561 +0000 UTC m=+3108.418816929" Oct 30 19:56:41.213996 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:41.213955 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.880073 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:41.880041 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.032159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.032114 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx"] Oct 30 19:56:42.041106 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.041067 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.041986 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.041965 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5250ee-xzmjx"] Oct 30 19:56:42.182473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.182369 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.407220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.407180 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ec35e5-62a3-4ee1-b73f-98b264bec799" path="/var/lib/kubelet/pods/21ec35e5-62a3-4ee1-b73f-98b264bec799/volumes" Oct 30 19:56:42.543969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:42.543863 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.058262 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:43.058223 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.201693 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:43.201627 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.993956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:43.993912 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.627740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:45.627705 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:47.446906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:47.446866 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.610732 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:48.610699 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 19:56:49.033426 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.033344 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.189127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.188711 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.271305 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.271255 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:56:49.271855 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.271799 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" containerID="cri-o://4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a" gracePeriod=30 Oct 30 19:56:49.355265 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.355231 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.604131 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:49.601530 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.500114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:51.499342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.662438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:51.662408 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:56:51.662615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:51.662482 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:56:53.115622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:53.115570 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.846327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:53.846288 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.563424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.563384 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.757618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.757589 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:56:54.794911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.794873 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.868322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.868291 4990 generic.go:356] "Generic (PLEG): container finished" podID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerID="4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a" exitCode=0 Oct 30 19:56:54.868540 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.868522 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" event={"ID":"b8cbf048-c3a9-40b3-b589-5844ba2e8baa","Type":"ContainerDied","Data":"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a"} Oct 30 19:56:54.868622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.868576 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" event={"ID":"b8cbf048-c3a9-40b3-b589-5844ba2e8baa","Type":"ContainerDied","Data":"1a0eed62589aff9c304d80a4cce884a0b752158a3e18e9ccfc34a64f46e3f467"} Oct 30 19:56:54.868622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.868610 4990 scope.go:122] "RemoveContainer" containerID="4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a" Oct 30 19:56:54.868780 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.868764 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv" Oct 30 19:56:54.881852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.881830 4990 scope.go:122] "RemoveContainer" containerID="4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a" Oct 30 19:56:54.882165 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:56:54.882143 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a\": container with ID starting with 4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a not found: ID does not exist" containerID="4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a" Oct 30 19:56:54.882246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.882174 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a"} err="failed to get container status \"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a\": rpc error: code = NotFound desc = could not find container \"4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a\": container with ID starting with 4356c8968f55236befbd058a4468fa6a0951849ebb691c67120ae8a2ce66a73a not found: ID does not exist" Oct 30 19:56:54.883663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.883640 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.883772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.883723 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.883846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.883773 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.883846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.883824 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.883956 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.883931 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.884033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884019 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.884192 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884177 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.884274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884231 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.884335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884261 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.884335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884276 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\") pod \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\" (UID: \"b8cbf048-c3a9-40b3-b589-5844ba2e8baa\") " Oct 30 19:56:54.885012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884616 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.885012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884806 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.885012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.884984 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:54.885277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.885165 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.885277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.885181 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.885277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.885190 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.885277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.885199 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.885735 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.885711 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.888069 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.888045 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:54.888575 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.888541 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.888661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.888607 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "kube-api-access-tsspx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:54.889553 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.889527 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token" pod "b8cbf048-c3a9-40b3-b589-5844ba2e8baa" (UID: "b8cbf048-c3a9-40b3-b589-5844ba2e8baa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:54.986848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.986820 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.986848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.986844 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.987021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.986854 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.987021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.986863 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.987021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:54.986872 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsspx\" (UniqueName: \"kubernetes.io/projected/b8cbf048-c3a9-40b3-b589-5844ba2e8baa-kube-api-access-tsspx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.028163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:55.028126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.215481 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:55.215449 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:56:55.225031 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:55.225007 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vjkqv"] Oct 30 19:56:56.403226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:56.403191 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" path="/var/lib/kubelet/pods/b8cbf048-c3a9-40b3-b589-5844ba2e8baa/volumes" Oct 30 19:56:56.646876 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:56.646841 4990 scope.go:122] "RemoveContainer" containerID="4ae72e8ed880db291f109303767d50e54c2784f00fdfdf5cbc3201f1d0e3bf19" Oct 30 19:56:56.658951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:56.658897 4990 scope.go:122] "RemoveContainer" containerID="c406ff64b73aadaff294c862749dfa5878312a0eb012a17cfb257605835b25e3" Oct 30 19:56:56.753918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:56.753868 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:57.029078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:57.029002 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg"] Oct 30 19:56:57.037795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:57.037768 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e514d1f8-vbwbg"] Oct 30 19:56:58.382271 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:58.382231 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:58.415598 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:58.415563 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da5f701-c757-4c29-8abb-cc5fed321176" path="/var/lib/kubelet/pods/4da5f701-c757-4c29-8abb-cc5fed321176/volumes" Oct 30 19:56:58.834068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:58.834030 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:56:58.834276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:58.834235 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" containerID="cri-o://85a6fcda4a155de9b2828256f3144ffe464ac0daacbcfc5cfb569d8b2793cc64" gracePeriod=30 Oct 30 19:56:59.664234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:59.664194 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:59.829134 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:56:59.828052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.149958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.149922 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.437191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.437081 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.454365 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.455563 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.455584 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.455867 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cbf048-c3a9-40b3-b589-5844ba2e8baa" containerName="istio-proxy" Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.462435 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 19:57:00.464814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.462526 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.474880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.474855 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2afc11\"" Oct 30 19:57:00.475611 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.475591 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2afc11-dockercfg-5dkjs\"" Oct 30 19:57:00.669115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.668335 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfdv\" (UniqueName: \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.669115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.668438 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.789251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.786525 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfdv\" (UniqueName: \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.789251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.786634 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.789251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.787998 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.832109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.814168 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfdv\" (UniqueName: \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\") pod \"payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:00.945352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:00.945304 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.101730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.101700 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:01.417802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.417775 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 19:57:01.418544 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:01.418516 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c170c74_2736_4d97_8b7d_c651fa404435.slice/crio-5417e206763d29fdb3d866738fb72751ab5fe97d7868b78d4230c929851dab81 WatchSource:0}: Error finding container 5417e206763d29fdb3d866738fb72751ab5fe97d7868b78d4230c929851dab81: Status 404 returned error can't find the container with id 5417e206763d29fdb3d866738fb72751ab5fe97d7868b78d4230c929851dab81 Oct 30 19:57:01.955379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.955346 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" event={"ID":"7c170c74-2736-4d97-8b7d-c651fa404435","Type":"ContainerStarted","Data":"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b"} Oct 30 19:57:01.955781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.955401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" event={"ID":"7c170c74-2736-4d97-8b7d-c651fa404435","Type":"ContainerStarted","Data":"5417e206763d29fdb3d866738fb72751ab5fe97d7868b78d4230c929851dab81"} Oct 30 19:57:01.955781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.955598 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:01.992572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:01.992516 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" podStartSLOduration=1.99250244 podStartE2EDuration="1.99250244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:01.988988215 +0000 UTC m=+3129.700513582" watchObservedRunningTime="2026-10-30 19:57:01.99250244 +0000 UTC m=+3129.704027851" Oct 30 19:57:02.640350 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:02.640314 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.066251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:03.066207 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.340414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:03.340338 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.491381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:03.491323 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.980199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:03.980163 4990 generic.go:356] "Generic (PLEG): container finished" podID="96e27524-6b26-4afc-92ef-32af2448c923" containerID="85a6fcda4a155de9b2828256f3144ffe464ac0daacbcfc5cfb569d8b2793cc64" exitCode=0 Oct 30 19:57:03.980452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:03.980310 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" event={"ID":"96e27524-6b26-4afc-92ef-32af2448c923","Type":"ContainerDied","Data":"85a6fcda4a155de9b2828256f3144ffe464ac0daacbcfc5cfb569d8b2793cc64"} Oct 30 19:57:04.130708 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.130676 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:57:04.238783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.238752 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.238963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.238821 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.238963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.238864 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.238963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.238905 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.239197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239007 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.239197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239048 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.239197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239125 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:04.239197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239149 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:04.239197 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239170 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.240671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239341 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:04.240671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239455 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.240671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239537 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\") pod \"96e27524-6b26-4afc-92ef-32af2448c923\" (UID: \"96e27524-6b26-4afc-92ef-32af2448c923\") " Oct 30 19:57:04.240671 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.239636 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:04.240935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.240666 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:04.240935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.240699 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.240935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.240723 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.240935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.240756 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.240935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.240770 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.253127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.253004 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "kube-api-access-srwvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:04.253127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.253065 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:04.253332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.253217 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:04.253332 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.253279 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo" pod "96e27524-6b26-4afc-92ef-32af2448c923" (UID: "96e27524-6b26-4afc-92ef-32af2448c923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:04.342579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.342545 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.342579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.342575 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e27524-6b26-4afc-92ef-32af2448c923-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.342777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.342586 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwvb\" (UniqueName: \"kubernetes.io/projected/96e27524-6b26-4afc-92ef-32af2448c923-kube-api-access-srwvb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.342777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.342598 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e27524-6b26-4afc-92ef-32af2448c923-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.342777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.342608 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e27524-6b26-4afc-92ef-32af2448c923-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:04.725556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.725469 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.999181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.993119 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" Oct 30 19:57:04.999181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.993116 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c" event={"ID":"96e27524-6b26-4afc-92ef-32af2448c923","Type":"ContainerDied","Data":"901d5c79a509da70b2bb05f9010457fd908d73c8b3763c6e3977cd1733b39f34"} Oct 30 19:57:04.999181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:04.993349 4990 scope.go:122] "RemoveContainer" containerID="85a6fcda4a155de9b2828256f3144ffe464ac0daacbcfc5cfb569d8b2793cc64" Oct 30 19:57:05.051024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:05.050993 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:57:05.060397 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:05.060368 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-mhg9c"] Oct 30 19:57:05.107476 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:05.107440 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:57:06.406220 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:06.406185 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e27524-6b26-4afc-92ef-32af2448c923" path="/var/lib/kubelet/pods/96e27524-6b26-4afc-92ef-32af2448c923/volumes" Oct 30 19:57:06.542411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:06.542379 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.614544 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:06.614498 4990 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7] : Timed out while waiting for systemd to remove kubepods-burstable-pod8bfd7eb6_16e0_4988_a6ca_a89811c3b0b7.slice" Oct 30 19:57:06.614714 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:06.614559 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7] : unable to destroy cgroup paths for cgroup [kubepods burstable pod8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7] : Timed out while waiting for systemd to remove kubepods-burstable-pod8bfd7eb6_16e0_4988_a6ca_a89811c3b0b7.slice" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" podUID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" Oct 30 19:57:06.704970 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:06.704878 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.016284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:07.016192 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm" Oct 30 19:57:08.027912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.027876 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5"] Oct 30 19:57:08.037172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.037147 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa33c00c-vbcf5"] Oct 30 19:57:08.279307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.279223 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.412935 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.412902 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad714d7d-6b6c-48c2-89d2-fd6ea684ab80" path="/var/lib/kubelet/pods/ad714d7d-6b6c-48c2-89d2-fd6ea684ab80/volumes" Oct 30 19:57:08.426063 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.426027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.713473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.713435 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:57:08.715170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.715077 4990 cpu_manager.go:414] "containerMap: removing container" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" Oct 30 19:57:08.715366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.715350 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" Oct 30 19:57:08.715748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.715729 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e27524-6b26-4afc-92ef-32af2448c923" containerName="istio-proxy" Oct 30 19:57:08.728872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.728846 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:57:08.729802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.729161 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.745172 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.745081 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e6016\"" Oct 30 19:57:08.745336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.745313 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e6016-dockercfg-r9cqg\"" Oct 30 19:57:08.846056 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.846018 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.846374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.846355 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p2m\" (UniqueName: \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.948361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.948328 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p2m\" (UniqueName: \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.948554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.948436 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.949202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.949177 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:08.974307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:08.974236 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p2m\" (UniqueName: \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\") pod \"payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:09.002846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:09.002807 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.055613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:09.055578 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:09.394008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:09.393978 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:57:09.397064 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:09.397034 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7561f0bd_baeb_4c7f_9870_f2e4dd833c5b.slice/crio-bc6c671109a0cbea141a2a8708043a150d82030af6270c8c4896379c4f1246b1 WatchSource:0}: Error finding container bc6c671109a0cbea141a2a8708043a150d82030af6270c8c4896379c4f1246b1: Status 404 returned error can't find the container with id bc6c671109a0cbea141a2a8708043a150d82030af6270c8c4896379c4f1246b1 Oct 30 19:57:10.046478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.046444 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" event={"ID":"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b","Type":"ContainerStarted","Data":"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086"} Oct 30 19:57:10.046661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.046500 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" event={"ID":"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b","Type":"ContainerStarted","Data":"bc6c671109a0cbea141a2a8708043a150d82030af6270c8c4896379c4f1246b1"} Oct 30 19:57:10.046661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.046531 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:10.090135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.090060 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" podStartSLOduration=2.090042409 podStartE2EDuration="2.090042409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:10.082173639 +0000 UTC m=+3137.793699011" watchObservedRunningTime="2026-10-30 19:57:10.090042409 +0000 UTC m=+3137.801567777" Oct 30 19:57:10.291281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.291244 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.494643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.494605 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.983606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.983572 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:57:10.988455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.988432 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:10.995531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.995500 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzn4l\" (UniqueName: \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:10.995817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:10.995776 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.007402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.007379 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:57:11.011061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.011034 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-59f9ed\"" Oct 30 19:57:11.011198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.011143 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-59f9ed-dockercfg-r7npr\"" Oct 30 19:57:11.097605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.097576 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.098118 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.097712 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzn4l\" (UniqueName: \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.098502 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.098476 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.115314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.115282 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzn4l\" (UniqueName: \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.306052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.306022 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:11.624958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.624929 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:57:11.625904 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:11.625879 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af99856_df61_42ee_961e_96d37703979f.slice/crio-f7e9cb75028a63f982db8e2f80375464be9acf460492b0b97b9dc277fb3bfb5a WatchSource:0}: Error finding container f7e9cb75028a63f982db8e2f80375464be9acf460492b0b97b9dc277fb3bfb5a: Status 404 returned error can't find the container with id f7e9cb75028a63f982db8e2f80375464be9acf460492b0b97b9dc277fb3bfb5a Oct 30 19:57:11.649082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:11.648950 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.068680 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:12.068642 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" event={"ID":"5af99856-df61-42ee-961e-96d37703979f","Type":"ContainerStarted","Data":"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c"} Oct 30 19:57:12.068905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:12.068712 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:12.068905 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:12.068739 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" event={"ID":"5af99856-df61-42ee-961e-96d37703979f","Type":"ContainerStarted","Data":"f7e9cb75028a63f982db8e2f80375464be9acf460492b0b97b9dc277fb3bfb5a"} Oct 30 19:57:12.119341 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:12.119283 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" podStartSLOduration=2.119268402 podStartE2EDuration="2.119268402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:12.11828808 +0000 UTC m=+3139.829813446" watchObservedRunningTime="2026-10-30 19:57:12.119268402 +0000 UTC m=+3139.830793767" Oct 30 19:57:12.967866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:12.967836 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 19:57:13.276888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:13.276803 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.474626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:13.474587 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.624142 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:13.624083 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.040717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:15.040683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.306851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:15.306813 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.661121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:16.661046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.334746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:18.334704 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.029941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:20.029905 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.061314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:21.061279 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:57:21.332614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:21.332508 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.481246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:21.481208 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.771765 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:21.771680 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.080179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:23.080136 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" podUID="5af99856-df61-42ee-961e-96d37703979f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.1.33:9004: connect: connection refused" Oct 30 19:57:23.969281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:23.969238 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.121963 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.121923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.234327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.234245 4990 generic.go:356] "Generic (PLEG): container finished" podID="9557ea28-d430-4305-b313-107486eb594b" containerID="9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081" exitCode=0 Oct 30 19:57:24.234327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.234320 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerDied","Data":"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081"} Oct 30 19:57:24.234859 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.234841 4990 scope.go:122] "RemoveContainer" containerID="9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081" Oct 30 19:57:24.236846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.236802 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:57:24.236967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.236886 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:57:24.236967 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:24.236916 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:57:25.263786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:25.263747 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerStarted","Data":"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b"} Oct 30 19:57:25.934461 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:25.933777 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.010918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:26.010885 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:57:26.011284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:26.011140 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 19:57:26.011491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:26.011457 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 19:57:27.011650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:27.011599 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 19:57:27.012186 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:27.011674 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 19:57:27.961737 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:27.959545 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.012061 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:28.012016 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 19:57:28.012499 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:28.012154 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 19:57:29.011424 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:29.011386 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 19:57:29.011604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:29.011457 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 19:57:30.010958 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:30.010921 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 19:57:30.011370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:30.010982 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 19:57:30.606478 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:30.606441 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:57:31.017459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:31.017375 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:57:31.369054 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:31.369027 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 19:57:31.945326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:31.945295 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.079170 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:32.079134 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.800791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:32.800752 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:57:32.801273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:32.801238 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" podUID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" containerName="maas-api" containerID="cri-o://2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe" gracePeriod=30 Oct 30 19:57:32.897911 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:32.897879 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.115658 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.115569 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:57:33.188648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.188618 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:57:33.239556 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.239523 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\" (UniqueName: \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\") pod \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " Oct 30 19:57:33.239723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.239633 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\") pod \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " Oct 30 19:57:33.239723 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.239684 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\") pod \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " Oct 30 19:57:33.239872 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.239721 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\") pod \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\" (UID: \"551cf2ae-f52e-4a69-8f13-039ecfa0ab65\") " Oct 30 19:57:33.241082 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.241049 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca" pod "551cf2ae-f52e-4a69-8f13-039ecfa0ab65" (UID: "551cf2ae-f52e-4a69-8f13-039ecfa0ab65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:33.242309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.242274 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg" pod "551cf2ae-f52e-4a69-8f13-039ecfa0ab65" (UID: "551cf2ae-f52e-4a69-8f13-039ecfa0ab65"). InnerVolumeSpecName "kube-api-access-ddfpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:33.242433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.242359 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs" pod "551cf2ae-f52e-4a69-8f13-039ecfa0ab65" (UID: "551cf2ae-f52e-4a69-8f13-039ecfa0ab65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:33.247486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.247456 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls" pod "551cf2ae-f52e-4a69-8f13-039ecfa0ab65" (UID: "551cf2ae-f52e-4a69-8f13-039ecfa0ab65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:33.342543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.342507 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddfpg\" (UniqueName: \"kubernetes.io/projected/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-kube-api-access-ddfpg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:33.342543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.342542 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:33.342719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.342556 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:33.342719 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.342565 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551cf2ae-f52e-4a69-8f13-039ecfa0ab65-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:33.468163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.456265 4990 generic.go:356] "Generic (PLEG): container finished" podID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" containerID="2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe" exitCode=0 Oct 30 19:57:33.468163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.456528 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" event={"ID":"551cf2ae-f52e-4a69-8f13-039ecfa0ab65","Type":"ContainerDied","Data":"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe"} Oct 30 19:57:33.468163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.456578 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" event={"ID":"551cf2ae-f52e-4a69-8f13-039ecfa0ab65","Type":"ContainerDied","Data":"0432f412b08d0c6daf62f7e1c44b1338684275146ef75096018fbc80c5df40e4"} Oct 30 19:57:33.468163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.456607 4990 scope.go:122] "RemoveContainer" containerID="2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe" Oct 30 19:57:33.468163 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.456771 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw" Oct 30 19:57:33.469632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.469598 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.496326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.496207 4990 scope.go:122] "RemoveContainer" containerID="2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe" Oct 30 19:57:33.496882 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:33.496789 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe\": container with ID starting with 2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe not found: ID does not exist" containerID="2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe" Oct 30 19:57:33.496882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.496839 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe"} err="failed to get container status \"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe\": rpc error: code = NotFound desc = could not find container \"2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe\": container with ID starting with 2d79eb2ff3083117dac73e44f99cb8c81effa574c72f8a653ae8c5526ab2aefe not found: ID does not exist" Oct 30 19:57:33.574201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.574134 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:57:33.584500 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.584293 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-aae4d8-6fbccd7c49-w22dw"] Oct 30 19:57:33.619259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:33.619224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.023748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.023714 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.163796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.163760 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:57:34.310296 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.310245 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.404769 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.404735 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" path="/var/lib/kubelet/pods/551cf2ae-f52e-4a69-8f13-039ecfa0ab65/volumes" Oct 30 19:57:34.446432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.446389 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.593626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:34.593533 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:35.105051 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:35.105017 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:57:35.557800 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:35.554910 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.212373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:37.210740 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.121320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.121289 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 19:57:38.122460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.122431 4990 cpu_manager.go:414] "containerMap: removing container" podUID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" containerName="maas-api" Oct 30 19:57:38.122460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.122458 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" containerName="maas-api" Oct 30 19:57:38.122635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.122617 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="551cf2ae-f52e-4a69-8f13-039ecfa0ab65" containerName="maas-api" Oct 30 19:57:38.126787 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.126768 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.147029 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.147005 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 19:57:38.149452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.149385 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed2ede\"/\"ipp-route-ed2ede-kserve-self-signed-certs\"" Oct 30 19:57:38.149452 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.149427 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed2ede\"/\"kube-root-ca.crt\"" Oct 30 19:57:38.149660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.149536 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ed2ede\"/\"openshift-service-ca.crt\"" Oct 30 19:57:38.149660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.149585 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ed2ede\"/\"default-dockercfg-b6mb8\"" Oct 30 19:57:38.150136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150044 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.150261 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150201 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.150377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150332 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.150462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150381 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fx5h\" (UniqueName: \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.150462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150415 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.150462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.150456 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.233663 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.233636 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.251752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.251729 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.251892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.251805 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.251953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.251914 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.251953 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.251949 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fx5h\" (UniqueName: \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.252060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.252015 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.252177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.252144 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.252251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.252193 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.252377 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.252347 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.252439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.252365 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.254308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.254284 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.254701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.254680 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.269655 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.269625 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fx5h\" (UniqueName: \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\") pod \"ipp-route-ed2ede-kserve-769456bcb5-pgngd\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.457078 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.456992 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:38.815325 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.815292 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 19:57:38.817739 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:38.817708 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09f6563c_411e_4018_b4e0_531f774367f0.slice/crio-d65ccddc32aba252a78616b63190b936ac5c95860f482358a352c0a5220419b6 WatchSource:0}: Error finding container d65ccddc32aba252a78616b63190b936ac5c95860f482358a352c0a5220419b6: Status 404 returned error can't find the container with id d65ccddc32aba252a78616b63190b936ac5c95860f482358a352c0a5220419b6 Oct 30 19:57:38.924152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:38.918290 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.164934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.164841 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" podUID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" containerName="payload-pre-processing" containerID="cri-o://acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a" gracePeriod=25 Oct 30 19:57:39.533351 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.533323 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:57:39.544647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.544611 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" event={"ID":"09f6563c-411e-4018-b4e0-531f774367f0","Type":"ContainerStarted","Data":"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e"} Oct 30 19:57:39.544788 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.544664 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" event={"ID":"09f6563c-411e-4018-b4e0-531f774367f0","Type":"ContainerStarted","Data":"d65ccddc32aba252a78616b63190b936ac5c95860f482358a352c0a5220419b6"} Oct 30 19:57:39.546576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.546549 4990 generic.go:356] "Generic (PLEG): container finished" podID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" containerID="acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a" exitCode=0 Oct 30 19:57:39.546726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.546590 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" Oct 30 19:57:39.546726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.546644 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" event={"ID":"3c2b499c-d5a1-4196-8971-bd5737df4ec7","Type":"ContainerDied","Data":"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a"} Oct 30 19:57:39.546851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.546737 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz" event={"ID":"3c2b499c-d5a1-4196-8971-bd5737df4ec7","Type":"ContainerDied","Data":"e82b6474b9c72a616dbae3d2af1c9f1ea9c73768ada3a8feb700af549ed03f3a"} Oct 30 19:57:39.546851 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.546751 4990 scope.go:122] "RemoveContainer" containerID="acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a" Oct 30 19:57:39.560272 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.560252 4990 scope.go:122] "RemoveContainer" containerID="acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a" Oct 30 19:57:39.560575 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:39.560555 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a\": container with ID starting with acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a not found: ID does not exist" containerID="acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a" Oct 30 19:57:39.560629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.560585 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a"} err="failed to get container status \"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a\": rpc error: code = NotFound desc = could not find container \"acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a\": container with ID starting with acd146bb176b364528a8b1702c475b8fd8429cb41bdafb08df62a1e0417af58a not found: ID does not exist" Oct 30 19:57:39.590897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.590346 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\" (UniqueName: \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\") pod \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " Oct 30 19:57:39.590897 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.590457 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\") pod \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\" (UID: \"3c2b499c-d5a1-4196-8971-bd5737df4ec7\") " Oct 30 19:57:39.592314 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.591991 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume" pod "3c2b499c-d5a1-4196-8971-bd5737df4ec7" (UID: "3c2b499c-d5a1-4196-8971-bd5737df4ec7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:39.595673 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.595623 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6" pod "3c2b499c-d5a1-4196-8971-bd5737df4ec7" (UID: "3c2b499c-d5a1-4196-8971-bd5737df4ec7"). InnerVolumeSpecName "kube-api-access-tt5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:39.634278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.634223 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" podStartSLOduration=1.6342061810000001 podStartE2EDuration="1.634206181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:39.629807772 +0000 UTC m=+3167.341333149" watchObservedRunningTime="2026-10-30 19:57:39.634206181 +0000 UTC m=+3167.345731548" Oct 30 19:57:39.692884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.692847 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5q6\" (UniqueName: \"kubernetes.io/projected/3c2b499c-d5a1-4196-8971-bd5737df4ec7-kube-api-access-tt5q6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:39.692884 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.692881 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c2b499c-d5a1-4196-8971-bd5737df4ec7-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:39.952751 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.951981 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:57:39.968294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:39.968258 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5457f96475-2crgz"] Oct 30 19:57:40.406581 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:40.406548 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" path="/var/lib/kubelet/pods/3c2b499c-d5a1-4196-8971-bd5737df4ec7/volumes" Oct 30 19:57:40.458086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:40.458057 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:40.866372 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:40.866304 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.146531 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.146444 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.331623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.331589 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.415199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.411972 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:57:41.415199 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.412226 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" containerID="cri-o://75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4" gracePeriod=30 Oct 30 19:57:41.493012 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.492980 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.666757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.666644 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.957049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:41.956952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.091181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.091143 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p"] Oct 30 19:57:42.091929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.091911 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" containerName="payload-pre-processing" Oct 30 19:57:42.091929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.091928 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" containerName="payload-pre-processing" Oct 30 19:57:42.092103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.092077 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2b499c-d5a1-4196-8971-bd5737df4ec7" containerName="payload-pre-processing" Oct 30 19:57:42.095748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.095723 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.109248 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.109220 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:42.116052 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.116028 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p"] Oct 30 19:57:42.119234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.119216 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:57:42.211977 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.211898 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rndl\" (UniqueName: \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.212159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.212024 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.212159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.212066 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.312959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.312929 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.313232 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.313211 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rndl\" (UniqueName: \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.313371 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.313329 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.313695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.313675 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.315791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.315764 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.331214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.331186 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rndl\" (UniqueName: \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\") pod \"maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.410361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.410331 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:42.639992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.639938 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.720802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:42.720774 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p"] Oct 30 19:57:42.721429 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:42.721398 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe57a030_1ce2_4853_8c3a_c6f1b4cb2405.slice/crio-d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2 WatchSource:0}: Error finding container d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2: Status 404 returned error can't find the container with id d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2 Oct 30 19:57:43.102715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:43.102681 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:57:43.103079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:43.102742 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:57:43.684280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:43.684237 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" event={"ID":"be57a030-1ce2-4853-8c3a-c6f1b4cb2405","Type":"ContainerStarted","Data":"2022dfd8430bd69feb86818e3fa440cf44a07017b4a130bf142842b6714534fc"} Oct 30 19:57:43.684508 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:43.684306 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" event={"ID":"be57a030-1ce2-4853-8c3a-c6f1b4cb2405","Type":"ContainerStarted","Data":"d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2"} Oct 30 19:57:43.724837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:43.724772 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" podStartSLOduration=1.7247541709999998 podStartE2EDuration="1.724754171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:43.719610624 +0000 UTC m=+3171.431136002" watchObservedRunningTime="2026-10-30 19:57:43.724754171 +0000 UTC m=+3171.436279539" Oct 30 19:57:44.643049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:44.643008 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.703652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:44.703611 4990 generic.go:356] "Generic (PLEG): container finished" podID="be57a030-1ce2-4853-8c3a-c6f1b4cb2405" containerID="2022dfd8430bd69feb86818e3fa440cf44a07017b4a130bf142842b6714534fc" exitCode=0 Oct 30 19:57:44.703875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:44.703708 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" event={"ID":"be57a030-1ce2-4853-8c3a-c6f1b4cb2405","Type":"ContainerDied","Data":"2022dfd8430bd69feb86818e3fa440cf44a07017b4a130bf142842b6714534fc"} Oct 30 19:57:45.906331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.906302 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:45.974997 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.974921 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\") pod \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " Oct 30 19:57:45.975356 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.975330 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\") pod \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " Oct 30 19:57:45.975815 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.975794 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\" (UniqueName: \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\") pod \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\" (UID: \"be57a030-1ce2-4853-8c3a-c6f1b4cb2405\") " Oct 30 19:57:45.977308 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.977283 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca" pod "be57a030-1ce2-4853-8c3a-c6f1b4cb2405" (UID: "be57a030-1ce2-4853-8c3a-c6f1b4cb2405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:45.978546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.978495 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca" pod "be57a030-1ce2-4853-8c3a-c6f1b4cb2405" (UID: "be57a030-1ce2-4853-8c3a-c6f1b4cb2405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:45.981857 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:45.981830 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl" pod "be57a030-1ce2-4853-8c3a-c6f1b4cb2405" (UID: "be57a030-1ce2-4853-8c3a-c6f1b4cb2405"). InnerVolumeSpecName "kube-api-access-8rndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:46.079157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.079130 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rndl\" (UniqueName: \"kubernetes.io/projected/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-kube-api-access-8rndl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.079157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.079156 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.079327 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.079166 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be57a030-1ce2-4853-8c3a-c6f1b4cb2405-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.377010 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.376968 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj"] Oct 30 19:57:46.378294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.378265 4990 cpu_manager.go:414] "containerMap: removing container" podUID="be57a030-1ce2-4853-8c3a-c6f1b4cb2405" containerName="revoke-keys" Oct 30 19:57:46.378294 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.378292 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be57a030-1ce2-4853-8c3a-c6f1b4cb2405" containerName="revoke-keys" Oct 30 19:57:46.378489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.378475 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="be57a030-1ce2-4853-8c3a-c6f1b4cb2405" containerName="revoke-keys" Oct 30 19:57:46.382169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.382148 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.411793 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.411767 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj"] Oct 30 19:57:46.485193 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.485152 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.485479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.485458 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.485705 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.485677 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.587303 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.587266 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.587466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.587358 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.587601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.587582 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.588373 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.588343 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.589783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.589760 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.605965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.605917 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\") pod \"maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.673336 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.673309 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:57:46.708175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.708142 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:46.727299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.727268 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" Oct 30 19:57:46.727299 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.727280 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p" event={"ID":"be57a030-1ce2-4853-8c3a-c6f1b4cb2405","Type":"ContainerDied","Data":"d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2"} Oct 30 19:57:46.727511 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.727344 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47613c6f8bd275fdd8d244658098c642709b9c0850db070fb9082ca20bec4c2" Oct 30 19:57:46.731207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.729837 4990 generic.go:356] "Generic (PLEG): container finished" podID="2dba8c98-73c7-4670-b719-7304469946ca" containerID="75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4" exitCode=0 Oct 30 19:57:46.731207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.729883 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerDied","Data":"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4"} Oct 30 19:57:46.731207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.729923 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" event={"ID":"2dba8c98-73c7-4670-b719-7304469946ca","Type":"ContainerDied","Data":"ebe6cc8fd8d11c4c1eeec38b5b43de62b78aa1f10c23e681153df0f9f020f6b0"} Oct 30 19:57:46.731207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.729950 4990 scope.go:122] "RemoveContainer" containerID="75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4" Oct 30 19:57:46.731207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.730138 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s" Oct 30 19:57:46.756892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.756753 4990 scope.go:122] "RemoveContainer" containerID="263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960" Oct 30 19:57:46.778059 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.777304 4990 scope.go:122] "RemoveContainer" containerID="75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4" Oct 30 19:57:46.778207 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:46.778159 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4\": container with ID starting with 75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4 not found: ID does not exist" containerID="75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4" Oct 30 19:57:46.778207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.778189 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4"} err="failed to get container status \"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4\": rpc error: code = NotFound desc = could not find container \"75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4\": container with ID starting with 75706d1ef02ef15185390f5aec42ece56da5e21ea0d1fb8228aab17bead6a6c4 not found: ID does not exist" Oct 30 19:57:46.778207 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.778209 4990 scope.go:122] "RemoveContainer" containerID="263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960" Oct 30 19:57:46.778876 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:57:46.778856 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960\": container with ID starting with 263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960 not found: ID does not exist" containerID="263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960" Oct 30 19:57:46.778971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.778879 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960"} err="failed to get container status \"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960\": rpc error: code = NotFound desc = could not find container \"263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960\": container with ID starting with 263a93d345f7f9b805ae0a259387f97957cb4135d974da55e36641154a9ab960 not found: ID does not exist" Oct 30 19:57:46.791003 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.790974 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791064 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791490 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791608 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791570 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791772 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791755 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791908 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.791998 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.791968 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.792068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.792029 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.792068 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.792063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\") pod \"2dba8c98-73c7-4670-b719-7304469946ca\" (UID: \"2dba8c98-73c7-4670-b719-7304469946ca\") " Oct 30 19:57:46.793554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.793530 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:46.793904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.793881 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:46.794459 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.794433 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:46.794934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.794901 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:46.795026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.795008 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:46.801666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.801639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:46.801830 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.801812 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:46.801929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.801913 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:46.803754 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.803727 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7" pod "2dba8c98-73c7-4670-b719-7304469946ca" (UID: "2dba8c98-73c7-4670-b719-7304469946ca"). InnerVolumeSpecName "kube-api-access-tmkh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:46.885251 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.885217 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:57:46.894895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894836 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.894895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894871 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.894895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894884 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.894895 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894897 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dba8c98-73c7-4670-b719-7304469946ca-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.895230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894911 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.895230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894923 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dba8c98-73c7-4670-b719-7304469946ca-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.895230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894935 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dba8c98-73c7-4670-b719-7304469946ca-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.895230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894949 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkh7\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-kube-api-access-tmkh7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:46.895230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:46.894961 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dba8c98-73c7-4670-b719-7304469946ca-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:47.019413 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.019344 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.059520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.059493 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj"] Oct 30 19:57:47.059862 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:47.059841 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a730dd_058b_4a4e_a02c_e03951e1c12b.slice/crio-c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2 WatchSource:0}: Error finding container c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2: Status 404 returned error can't find the container with id c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2 Oct 30 19:57:47.081144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.080978 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:57:47.089166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.089137 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-aae4d8-openshift-default-b6bf8c499-p978s"] Oct 30 19:57:47.236900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.236860 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.741762 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.741724 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" event={"ID":"f6a730dd-058b-4a4e-a02c-e03951e1c12b","Type":"ContainerStarted","Data":"de881739e67cf33e72e2d0d52b2a6e8ed9a454f8a16db94b025e243a44894b14"} Oct 30 19:57:47.741981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.741773 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" event={"ID":"f6a730dd-058b-4a4e-a02c-e03951e1c12b","Type":"ContainerStarted","Data":"c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2"} Oct 30 19:57:47.783236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:47.783180 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" podStartSLOduration=1.7831619399999998 podStartE2EDuration="1.78316194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:47.776196864 +0000 UTC m=+3175.487722230" watchObservedRunningTime="2026-10-30 19:57:47.78316194 +0000 UTC m=+3175.494687307" Oct 30 19:57:48.406616 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.406580 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dba8c98-73c7-4670-b719-7304469946ca" path="/var/lib/kubelet/pods/2dba8c98-73c7-4670-b719-7304469946ca/volumes" Oct 30 19:57:48.457960 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.457931 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:48.471175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.471147 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:48.752454 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.752363 4990 generic.go:356] "Generic (PLEG): container finished" podID="f6a730dd-058b-4a4e-a02c-e03951e1c12b" containerID="de881739e67cf33e72e2d0d52b2a6e8ed9a454f8a16db94b025e243a44894b14" exitCode=0 Oct 30 19:57:48.752613 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.752510 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" event={"ID":"f6a730dd-058b-4a4e-a02c-e03951e1c12b","Type":"ContainerDied","Data":"de881739e67cf33e72e2d0d52b2a6e8ed9a454f8a16db94b025e243a44894b14"} Oct 30 19:57:48.765528 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:48.765505 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 19:57:50.016879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.016842 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.051168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.049051 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:50.156727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.156682 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\") pod \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " Oct 30 19:57:50.157605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.157436 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca" pod "f6a730dd-058b-4a4e-a02c-e03951e1c12b" (UID: "f6a730dd-058b-4a4e-a02c-e03951e1c12b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:50.158108 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.158071 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\") pod \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " Oct 30 19:57:50.158886 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.158862 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\") pod \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\" (UID: \"f6a730dd-058b-4a4e-a02c-e03951e1c12b\") " Oct 30 19:57:50.160046 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.160014 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:50.162519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.162490 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca" pod "f6a730dd-058b-4a4e-a02c-e03951e1c12b" (UID: "f6a730dd-058b-4a4e-a02c-e03951e1c12b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:50.162635 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.162586 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr" pod "f6a730dd-058b-4a4e-a02c-e03951e1c12b" (UID: "f6a730dd-058b-4a4e-a02c-e03951e1c12b"). InnerVolumeSpecName "kube-api-access-knrrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:50.261652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.261613 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrrr\" (UniqueName: \"kubernetes.io/projected/f6a730dd-058b-4a4e-a02c-e03951e1c12b-kube-api-access-knrrr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:50.261652 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.261654 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a730dd-058b-4a4e-a02c-e03951e1c12b-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:57:50.858326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.857081 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" event={"ID":"f6a730dd-058b-4a4e-a02c-e03951e1c12b","Type":"ContainerDied","Data":"c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2"} Oct 30 19:57:50.858326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.857167 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ed703124f41b48168c494a2731e77bf6faf74ff169c53422405c7ca79a22f2" Oct 30 19:57:50.858326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:50.857235 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj" Oct 30 19:57:51.338115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:51.337613 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.073733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:52.073694 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.656615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:52.656580 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:57:52.715435 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:52.715400 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:57:52.864741 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:52.864703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.019609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.019515 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.710609 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.710576 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 19:57:53.711386 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711370 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711387 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711400 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f6a730dd-058b-4a4e-a02c-e03951e1c12b" containerName="revoke-keys" Oct 30 19:57:53.711450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711406 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a730dd-058b-4a4e-a02c-e03951e1c12b" containerName="revoke-keys" Oct 30 19:57:53.711450 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711443 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711452 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711583 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711596 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dba8c98-73c7-4670-b719-7304469946ca" containerName="istio-proxy" Oct 30 19:57:53.711660 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.711604 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a730dd-058b-4a4e-a02c-e03951e1c12b" containerName="revoke-keys" Oct 30 19:57:53.715600 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.715575 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.736798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.736767 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 19:57:53.784116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.764916 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnpj\" (UniqueName: \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.784116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.765045 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.784420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.784391 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6e6016-serving-cert\"" Oct 30 19:57:53.795115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.789427 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.798711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.797155 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.898529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.898496 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnpj\" (UniqueName: \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.898720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.898577 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.898791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.898757 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.898941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.898924 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.899706 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.899672 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.901588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.901548 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.901798 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.901777 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.962229 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.957891 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnpj\" (UniqueName: \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\") pod \"maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:53.997979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:53.997938 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.031110 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.031064 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:54.163428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.162685 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.401433 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:57:54.401403 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd456f4c8_a2f1_4ad3_8f6e_19ac98065606.slice/crio-c1ad22e4a0c406dde6cc100a9e5248ecad34c07304ad4265a06832f3c5b20699 WatchSource:0}: Error finding container c1ad22e4a0c406dde6cc100a9e5248ecad34c07304ad4265a06832f3c5b20699: Status 404 returned error can't find the container with id c1ad22e4a0c406dde6cc100a9e5248ecad34c07304ad4265a06832f3c5b20699 Oct 30 19:57:54.408882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.408857 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 19:57:54.922318 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.922282 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" event={"ID":"d456f4c8-a2f1-4ad3-8f6e-19ac98065606","Type":"ContainerStarted","Data":"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43"} Oct 30 19:57:54.922717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.922345 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" event={"ID":"d456f4c8-a2f1-4ad3-8f6e-19ac98065606","Type":"ContainerStarted","Data":"c1ad22e4a0c406dde6cc100a9e5248ecad34c07304ad4265a06832f3c5b20699"} Oct 30 19:57:54.922717 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.922377 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:57:54.973567 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:54.973459 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" podStartSLOduration=1.659158583 podStartE2EDuration="1.973439405s" totalImagesPullingTime="314.280822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:54.960142269 +0000 UTC m=+3182.671667635" watchObservedRunningTime="2026-10-30 19:57:54.973439405 +0000 UTC m=+3182.684964773" Oct 30 19:57:55.845026 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:55.840767 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.797867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:56.797833 4990 scope.go:122] "RemoveContainer" containerID="89bedba5fb9f3e7c3d7aa3a95f9e0a128bf6e41e3a808f4cdb47e8da44120183" Oct 30 19:57:56.809410 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:56.809386 4990 scope.go:122] "RemoveContainer" containerID="a92d5080a6ffe6e36a39ba8f0691f0dd40a8c4c5fb819b91d53e1673c4828283" Oct 30 19:57:56.821141 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:56.821119 4990 scope.go:122] "RemoveContainer" containerID="db4c4ede185fb590d47298fe3cdb72b53b44a830a6d4d74715f0981f9be7d895" Oct 30 19:57:57.496021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:57.495889 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.657425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:57.657385 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" containerID="cri-o://b6bdd85d26435592faf0b5ef8854bbc7475f84f4a96685b305b9084b162c65ef" gracePeriod=25 Oct 30 19:57:57.716446 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:57.716384 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" containerID="cri-o://16fff3d24019cf12243c1b631e89cddb58cd34ac0f63bdd8fa83644cc8519dfc" gracePeriod=25 Oct 30 19:57:59.560474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:57:59.559052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.938934 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.938911 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 19:58:00.986443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.986417 4990 generic.go:356] "Generic (PLEG): container finished" podID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerID="bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b" exitCode=0 Oct 30 19:58:00.986582 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.986455 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerDied","Data":"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b"} Oct 30 19:58:00.986819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.986794 4990 scope.go:122] "RemoveContainer" containerID="bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b" Oct 30 19:58:00.988580 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.988548 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:00.988690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.988616 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:00.988690 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:00.988664 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:01.199202 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:01.196276 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.921650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:01.921601 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" containerID="cri-o://853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6" gracePeriod=45 Oct 30 19:58:02.006576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.006542 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerStarted","Data":"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec"} Oct 30 19:58:02.336066 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.335648 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:58:02.435814 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.435778 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:58:02.436000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.435816 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:58:02.436000 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.435864 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 19:58:02.459259 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.459226 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\" (UniqueName: \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.459489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.459467 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.459648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.459631 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.460484 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.460025 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.461375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.460986 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.461727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.461345 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.461847 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.461820 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.461975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.461956 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\") pod \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\" (UID: \"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5\") " Oct 30 19:58:02.462282 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.462247 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.462874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.462848 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.462959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.462879 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.462959 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.462892 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.520629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.513534 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:02.521023 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.520967 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.523746 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.523517 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk" pod "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" (UID: "8ba216e3-c6dd-4ca2-a4b3-39401f1921d5"). InnerVolumeSpecName "kube-api-access-2tbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:02.564999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.564918 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.564999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.564952 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbfk\" (UniqueName: \"kubernetes.io/projected/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-kube-api-access-2tbfk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.564999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.564967 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.752462 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.752416 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:02.900144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:02.898970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.020495 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.020455 4990 generic.go:356] "Generic (PLEG): container finished" podID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerID="853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6" exitCode=0 Oct 30 19:58:03.020993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.020502 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" Oct 30 19:58:03.020993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.020542 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" event={"ID":"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5","Type":"ContainerDied","Data":"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6"} Oct 30 19:58:03.020993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.020601 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9" event={"ID":"8ba216e3-c6dd-4ca2-a4b3-39401f1921d5","Type":"ContainerDied","Data":"39cf9b569d72b79f60849d278ac29922271fd74523d3f6984eb9bbec3ba11675"} Oct 30 19:58:03.020993 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.020628 4990 scope.go:122] "RemoveContainer" containerID="853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6" Oct 30 19:58:03.033116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.033076 4990 scope.go:122] "RemoveContainer" containerID="853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6" Oct 30 19:58:03.033396 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:03.033372 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6\": container with ID starting with 853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6 not found: ID does not exist" containerID="853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6" Oct 30 19:58:03.033473 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.033406 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6"} err="failed to get container status \"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6\": rpc error: code = NotFound desc = could not find container \"853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6\": container with ID starting with 853999464c52266bda9853e0df37c29c27333b01b8b048d780638d3650faa7f6 not found: ID does not exist" Oct 30 19:58:03.090117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.086669 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:58:03.102020 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.101988 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/premium-w6-6e0c55-kserve-bfc4db698-jltp9"] Oct 30 19:58:03.436417 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.436373 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:58:03.436604 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:03.436435 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 19:58:04.410983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:04.410937 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" path="/var/lib/kubelet/pods/8ba216e3-c6dd-4ca2-a4b3-39401f1921d5/volumes" Oct 30 19:58:04.436603 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:04.436576 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:58:04.436785 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:04.436622 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 19:58:04.526112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:04.525181 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.604111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:04.603221 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:05.119727 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:05.119690 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:58:05.436060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:05.435977 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:58:05.436060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:05.436043 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 19:58:05.603808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:05.603760 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:06.089724 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:06.089686 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:06.436335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:06.436259 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:58:06.436335 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:06.436309 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 19:58:07.441400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:07.441373 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:58:07.441791 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:07.441657 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:58:07.442467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:07.442448 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 19:58:07.703136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:07.703009 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.450230 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:09.450179 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.581969 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:09.581922 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.556388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:11.556348 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.899864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:12.899826 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.693401 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.693357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.830945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.830913 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:58:13.869021 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.868982 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm"] Oct 30 19:58:13.874532 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.874501 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-249dd4-7bhpm"] Oct 30 19:58:13.876375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.876341 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.883707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:13.883685 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:58:14.032316 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:14.032234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.170586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:14.170540 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.405133 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:14.405080 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7" path="/var/lib/kubelet/pods/8bfd7eb6-16e0-4988-a6ca-a89811c3b0b7/volumes" Oct 30 19:58:14.405975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:14.405944 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.599625 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:14.599586 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:15.603578 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:15.603541 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:16.463562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:16.463520 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.785518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:17.785482 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.967164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:17.952018 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:18.259165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.258355 4990 generic.go:356] "Generic (PLEG): container finished" podID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerID="16fff3d24019cf12243c1b631e89cddb58cd34ac0f63bdd8fa83644cc8519dfc" exitCode=0 Oct 30 19:58:18.259165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.258495 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" event={"ID":"047938d4-8ce7-42d8-af9a-0b909bc73524","Type":"ContainerDied","Data":"16fff3d24019cf12243c1b631e89cddb58cd34ac0f63bdd8fa83644cc8519dfc"} Oct 30 19:58:18.284116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.260808 4990 generic.go:356] "Generic (PLEG): container finished" podID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerID="b6bdd85d26435592faf0b5ef8854bbc7475f84f4a96685b305b9084b162c65ef" exitCode=0 Oct 30 19:58:18.284116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.260974 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" event={"ID":"e9ca7004-923b-4f87-80de-4481a7e8f182","Type":"ContainerDied","Data":"b6bdd85d26435592faf0b5ef8854bbc7475f84f4a96685b305b9084b162c65ef"} Oct 30 19:58:18.349539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.349313 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:58:18.354856 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.354830 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:58:18.494215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494179 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\") pod \"047938d4-8ce7-42d8-af9a-0b909bc73524\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " Oct 30 19:58:18.494384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494296 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\") pod \"e9ca7004-923b-4f87-80de-4481a7e8f182\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " Oct 30 19:58:18.494384 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494337 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\" (UniqueName: \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\") pod \"e9ca7004-923b-4f87-80de-4481a7e8f182\" (UID: \"e9ca7004-923b-4f87-80de-4481a7e8f182\") " Oct 30 19:58:18.494566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494549 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\" (UniqueName: \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\") pod \"047938d4-8ce7-42d8-af9a-0b909bc73524\" (UID: \"047938d4-8ce7-42d8-af9a-0b909bc73524\") " Oct 30 19:58:18.494641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494594 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume" pod "047938d4-8ce7-42d8-af9a-0b909bc73524" (UID: "047938d4-8ce7-42d8-af9a-0b909bc73524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:18.494713 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.494647 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume" pod "e9ca7004-923b-4f87-80de-4481a7e8f182" (UID: "e9ca7004-923b-4f87-80de-4481a7e8f182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:18.495651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.495625 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047938d4-8ce7-42d8-af9a-0b909bc73524-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:18.495789 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.495654 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ca7004-923b-4f87-80de-4481a7e8f182-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:18.496757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.496727 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d" pod "047938d4-8ce7-42d8-af9a-0b909bc73524" (UID: "047938d4-8ce7-42d8-af9a-0b909bc73524"). InnerVolumeSpecName "kube-api-access-ts84d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:18.496983 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.496967 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v" pod "e9ca7004-923b-4f87-80de-4481a7e8f182" (UID: "e9ca7004-923b-4f87-80de-4481a7e8f182"). InnerVolumeSpecName "kube-api-access-q986v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:18.597181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.597159 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q986v\" (UniqueName: \"kubernetes.io/projected/e9ca7004-923b-4f87-80de-4481a7e8f182-kube-api-access-q986v\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:18.597181 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.597182 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts84d\" (UniqueName: \"kubernetes.io/projected/047938d4-8ce7-42d8-af9a-0b909bc73524-kube-api-access-ts84d\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:18.831455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.831414 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" podUID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" containerName="payload-processing" containerID="cri-o://76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0" gracePeriod=25 Oct 30 19:58:18.880799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.880761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:18.884349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:18.884167 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" podUID="d2c84065-e853-41c1-b85b-f442561dee8e" containerName="payload-pre-processing" containerID="cri-o://75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69" gracePeriod=25 Oct 30 19:58:19.203281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.203244 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:58:19.217645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.217618 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:58:19.276210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.275889 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" event={"ID":"047938d4-8ce7-42d8-af9a-0b909bc73524","Type":"ContainerDied","Data":"0ef820468b3a880f5cb64e47ebe18f7f2ce50829bf06626b4405b0c68207929c"} Oct 30 19:58:19.276210 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.275988 4990 scope.go:122] "RemoveContainer" containerID="16fff3d24019cf12243c1b631e89cddb58cd34ac0f63bdd8fa83644cc8519dfc" Oct 30 19:58:19.276677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.276607 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct" Oct 30 19:58:19.284267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.279404 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" event={"ID":"e9ca7004-923b-4f87-80de-4481a7e8f182","Type":"ContainerDied","Data":"fbf584d877ae46a0bc09dc702f6f3ac453038175dc6d1820a07d0a1837339791"} Oct 30 19:58:19.284267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.279511 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr" Oct 30 19:58:19.287548 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.287517 4990 generic.go:356] "Generic (PLEG): container finished" podID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" containerID="76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0" exitCode=0 Oct 30 19:58:19.287670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.287601 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" event={"ID":"f31826d8-c296-4c75-a2a4-99a1f1c93f95","Type":"ContainerDied","Data":"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0"} Oct 30 19:58:19.287670 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.287644 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" event={"ID":"f31826d8-c296-4c75-a2a4-99a1f1c93f95","Type":"ContainerDied","Data":"91d69fd3edba2db0f3274d6ef9bd09b8b58677cc12876ee1cdd3a306af5536d8"} Oct 30 19:58:19.287807 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.287717 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6" Oct 30 19:58:19.290839 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.290794 4990 generic.go:356] "Generic (PLEG): container finished" podID="d2c84065-e853-41c1-b85b-f442561dee8e" containerID="75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69" exitCode=0 Oct 30 19:58:19.290981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.290964 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" event={"ID":"d2c84065-e853-41c1-b85b-f442561dee8e","Type":"ContainerDied","Data":"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69"} Oct 30 19:58:19.291064 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.291006 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" event={"ID":"d2c84065-e853-41c1-b85b-f442561dee8e","Type":"ContainerDied","Data":"e0313796c331019bb99ea1dbdd59ea1192cf5adf8f970dece16d5a8a2a482daa"} Oct 30 19:58:19.291165 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.291069 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp" Oct 30 19:58:19.304666 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.304649 4990 scope.go:122] "RemoveContainer" containerID="b6bdd85d26435592faf0b5ef8854bbc7475f84f4a96685b305b9084b162c65ef" Oct 30 19:58:19.308981 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.308961 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\") pod \"d2c84065-e853-41c1-b85b-f442561dee8e\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " Oct 30 19:58:19.309595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.309577 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\") pod \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " Oct 30 19:58:19.310270 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.310252 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\" (UniqueName: \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\") pod \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\" (UID: \"f31826d8-c296-4c75-a2a4-99a1f1c93f95\") " Oct 30 19:58:19.310733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.309505 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume" pod "d2c84065-e853-41c1-b85b-f442561dee8e" (UID: "d2c84065-e853-41c1-b85b-f442561dee8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:19.310733 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.310156 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume" pod "f31826d8-c296-4c75-a2a4-99a1f1c93f95" (UID: "f31826d8-c296-4c75-a2a4-99a1f1c93f95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:19.311076 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.311061 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\" (UniqueName: \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\") pod \"d2c84065-e853-41c1-b85b-f442561dee8e\" (UID: \"d2c84065-e853-41c1-b85b-f442561dee8e\") " Oct 30 19:58:19.312310 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.312288 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31826d8-c296-4c75-a2a4-99a1f1c93f95-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:19.312455 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.312441 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2c84065-e853-41c1-b85b-f442561dee8e-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:19.313058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.313038 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms" pod "f31826d8-c296-4c75-a2a4-99a1f1c93f95" (UID: "f31826d8-c296-4c75-a2a4-99a1f1c93f95"). InnerVolumeSpecName "kube-api-access-z4cms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:19.319266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.319228 4990 scope.go:122] "RemoveContainer" containerID="76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0" Oct 30 19:58:19.319588 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.319563 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z" pod "d2c84065-e853-41c1-b85b-f442561dee8e" (UID: "d2c84065-e853-41c1-b85b-f442561dee8e"). InnerVolumeSpecName "kube-api-access-8sb5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:19.333293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.333276 4990 scope.go:122] "RemoveContainer" containerID="76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0" Oct 30 19:58:19.333706 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:19.333672 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0\": container with ID starting with 76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0 not found: ID does not exist" containerID="76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0" Oct 30 19:58:19.333784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.333715 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0"} err="failed to get container status \"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0\": rpc error: code = NotFound desc = could not find container \"76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0\": container with ID starting with 76c184d3ecec1abee19994b0f890d2cc951864de4fcea13317596ff0254fe3a0 not found: ID does not exist" Oct 30 19:58:19.333784 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.333737 4990 scope.go:122] "RemoveContainer" containerID="75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69" Oct 30 19:58:19.345605 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.345590 4990 scope.go:122] "RemoveContainer" containerID="75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69" Oct 30 19:58:19.345853 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:19.345836 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69\": container with ID starting with 75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69 not found: ID does not exist" containerID="75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69" Oct 30 19:58:19.345918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.345861 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69"} err="failed to get container status \"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69\": rpc error: code = NotFound desc = could not find container \"75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69\": container with ID starting with 75cd6ff06538252c3c4213e1d60de63130262eb6fb0c59dd4222162043798a69 not found: ID does not exist" Oct 30 19:58:19.352647 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.352627 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:58:19.357529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.357508 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-6e0c55-675c7c4947-jslct"] Oct 30 19:58:19.401330 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.401309 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:58:19.407972 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.407909 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-6e0c55-6f85f69bdb-l84fr"] Oct 30 19:58:19.414367 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.414345 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4cms\" (UniqueName: \"kubernetes.io/projected/f31826d8-c296-4c75-a2a4-99a1f1c93f95-kube-api-access-z4cms\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:19.414458 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.414369 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sb5z\" (UniqueName: \"kubernetes.io/projected/d2c84065-e853-41c1-b85b-f442561dee8e-kube-api-access-8sb5z\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:19.632799 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.632770 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:58:19.646546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.646522 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-55965b8c7d-9tzw6"] Oct 30 19:58:19.679808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.679745 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:58:19.687301 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:19.687277 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-5695d8f5b7-4zmrp"] Oct 30 19:58:20.405483 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.405451 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" path="/var/lib/kubelet/pods/047938d4-8ce7-42d8-af9a-0b909bc73524/volumes" Oct 30 19:58:20.406044 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.406026 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c84065-e853-41c1-b85b-f442561dee8e" path="/var/lib/kubelet/pods/d2c84065-e853-41c1-b85b-f442561dee8e/volumes" Oct 30 19:58:20.406519 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.406504 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" path="/var/lib/kubelet/pods/e9ca7004-923b-4f87-80de-4481a7e8f182/volumes" Oct 30 19:58:20.406929 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.406917 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" path="/var/lib/kubelet/pods/f31826d8-c296-4c75-a2a4-99a1f1c93f95/volumes" Oct 30 19:58:20.749212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.749121 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.905381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:20.905345 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:21.029550 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:21.029301 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:58:21.029727 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:21.029550 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" containerID="cri-o://2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82" gracePeriod=30 Oct 30 19:58:21.109944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:21.109902 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.476908 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:22.473564 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.105352 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:24.105316 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.262116 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:24.261991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.811321 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:24.811285 4990 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:24.811543 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:24.811344 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:24.855975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:24.855940 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:25.950661 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:25.950622 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:26.336626 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.336592 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:58:26.363766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.363731 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.363951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.363786 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.363951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.363845 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.363951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.363917 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364028 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364125 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364172 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364236 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364234 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364313 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\") pod \"4a8ac718-7380-4a19-882e-ed1d63832f4b\" (UID: \"4a8ac718-7380-4a19-882e-ed1d63832f4b\") " Oct 30 19:58:26.364457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364371 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364726 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364921 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.364936 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365012 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365159 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365172 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365181 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a8ac718-7380-4a19-882e-ed1d63832f4b-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365189 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.365215 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.365197 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.367861 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.367839 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:26.368443 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368412 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "kube-api-access-8j4vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:26.368549 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368467 4990 generic.go:356] "Generic (PLEG): container finished" podID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerID="2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82" exitCode=0 Oct 30 19:58:26.368607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368580 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" Oct 30 19:58:26.368720 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368700 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" event={"ID":"4a8ac718-7380-4a19-882e-ed1d63832f4b","Type":"ContainerDied","Data":"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82"} Oct 30 19:58:26.368806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368756 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s" event={"ID":"4a8ac718-7380-4a19-882e-ed1d63832f4b","Type":"ContainerDied","Data":"b634459f4fffa3ee41798fb31c9e71ac8ce45ea4bc58d4638375dff9934ed28e"} Oct 30 19:58:26.368930 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.368837 4990 scope.go:122] "RemoveContainer" containerID="2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82" Oct 30 19:58:26.369311 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.369274 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:26.369586 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.369561 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy" pod "4a8ac718-7380-4a19-882e-ed1d63832f4b" (UID: "4a8ac718-7380-4a19-882e-ed1d63832f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.424246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.423884 4990 scope.go:122] "RemoveContainer" containerID="2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82" Oct 30 19:58:26.424595 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:26.424489 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82\": container with ID starting with 2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82 not found: ID does not exist" containerID="2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82" Oct 30 19:58:26.424595 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.424530 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82"} err="failed to get container status \"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82\": rpc error: code = NotFound desc = could not find container \"2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82\": container with ID starting with 2a19bca7bf4b74ffe0753901529461f8bbb4fd8b1f4ab971d50ac01416a67e82 not found: ID does not exist" Oct 30 19:58:26.468306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.467211 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.468306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.467246 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4vk\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-kube-api-access-8j4vk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.468306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.467259 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.468306 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.467273 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a8ac718-7380-4a19-882e-ed1d63832f4b-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.708748 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.708709 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:58:26.718277 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:26.718236 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-rmk5s"] Oct 30 19:58:27.625208 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:27.625168 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:28.403388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:28.403346 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" path="/var/lib/kubelet/pods/4a8ac718-7380-4a19-882e-ed1d63832f4b/volumes" Oct 30 19:58:29.335322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:29.335283 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.632414 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.632379 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.728319 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.728285 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q"] Oct 30 19:58:30.729185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729165 4990 cpu_manager.go:414] "containerMap: removing container" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" Oct 30 19:58:30.729185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729185 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729210 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729216 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729228 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" containerName="payload-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729234 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" containerName="payload-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729243 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729248 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729266 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729271 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729278 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d2c84065-e853-41c1-b85b-f442561dee8e" containerName="payload-pre-processing" Oct 30 19:58:30.729326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729283 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c84065-e853-41c1-b85b-f442561dee8e" containerName="payload-pre-processing" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729408 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ca7004-923b-4f87-80de-4481a7e8f182" containerName="payload-processing" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729422 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba216e3-c6dd-4ca2-a4b3-39401f1921d5" containerName="main" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729429 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c84065-e853-41c1-b85b-f442561dee8e" containerName="payload-pre-processing" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729441 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="047938d4-8ce7-42d8-af9a-0b909bc73524" containerName="payload-pre-processing" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729450 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8ac718-7380-4a19-882e-ed1d63832f4b" containerName="istio-proxy" Oct 30 19:58:30.729656 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.729455 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31826d8-c296-4c75-a2a4-99a1f1c93f95" containerName="payload-processing" Oct 30 19:58:30.734783 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.734758 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.751683 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.751659 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q"] Oct 30 19:58:30.757225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.757003 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:30.757225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.757069 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:58:30.800411 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.796234 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.841375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.841342 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhjb\" (UniqueName: \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.841574 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.841445 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.841684 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.841654 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.943874 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.943796 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.944037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.943965 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhjb\" (UniqueName: \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.944037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.944029 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.944797 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.944774 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.946467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.946445 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:30.961541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:30.961521 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhjb\" (UniqueName: \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\") pod \"maas-api-revoke-sub-65b8f637-92b2q\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:31.051878 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:31.051854 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:31.189601 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:31.187312 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:31.360672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:31.360647 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q"] Oct 30 19:58:31.361985 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:58:31.361961 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777f2108_a13f_4a40_b736_41b50534b7c2.slice/crio-4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428 WatchSource:0}: Error finding container 4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428: Status 404 returned error can't find the container with id 4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428 Oct 30 19:58:31.416615 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:31.416580 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" event={"ID":"777f2108-a13f-4a40-b736-41b50534b7c2","Type":"ContainerStarted","Data":"4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428"} Oct 30 19:58:32.307300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.307264 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:32.426866 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.426837 4990 generic.go:356] "Generic (PLEG): container finished" podID="777f2108-a13f-4a40-b736-41b50534b7c2" containerID="2584cfa0c06fd3f5a1030362f35d088e46a81841ee41a974cfc371f3c443280f" exitCode=0 Oct 30 19:58:32.427036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.426905 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" event={"ID":"777f2108-a13f-4a40-b736-41b50534b7c2","Type":"ContainerDied","Data":"2584cfa0c06fd3f5a1030362f35d088e46a81841ee41a974cfc371f3c443280f"} Oct 30 19:58:32.636369 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.636333 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 19:58:32.641354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.641324 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.661899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.661858 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e430a0-openshift-default-dockercfg-nkztw\"" Oct 30 19:58:32.669541 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.669512 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 19:58:32.735730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.735698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774122 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774304 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774367 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774431 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774484 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774542 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774576 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774620 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.777278 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.774644 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffm56\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.876576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.876537 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.896238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.896154 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffm56\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.905241 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.905208 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.906379 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.906269 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.906707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.906687 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.914119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.914080 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.914697 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.914678 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.926573 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.926547 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.927137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.927117 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.928965 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.928946 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.929852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.927006 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.947176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.928885 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.947176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.914585 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.947176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.929697 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.954178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.949682 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.954178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.950239 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.954178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.950892 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffm56\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.954178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.953733 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\") pod \"e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:32.962153 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:32.958953 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:33.236019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.235934 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.298979 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.298780 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 19:58:33.302323 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.302271 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:33.302460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.302373 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:33.302460 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.302417 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:58:33.404631 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.404598 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.440638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.440604 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerStarted","Data":"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d"} Oct 30 19:58:33.440802 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.440669 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerStarted","Data":"a31d68bddac37ebf19311fa38a0ae0f2707bbfcceaf84938dddf6fe7acacf92c"} Oct 30 19:58:33.485643 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.485571 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podStartSLOduration=1.485550725 podStartE2EDuration="1.485550725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:33.483443369 +0000 UTC m=+3221.194968736" watchObservedRunningTime="2026-10-30 19:58:33.485550725 +0000 UTC m=+3221.197076092" Oct 30 19:58:33.507912 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.507816 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:58:33.508694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.508430 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" podUID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" containerName="payload-pre-processing" containerID="cri-o://e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d" gracePeriod=30 Oct 30 19:58:33.611599 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.611413 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.717206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.717174 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:33.745881 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.745845 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\") pod \"777f2108-a13f-4a40-b736-41b50534b7c2\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " Oct 30 19:58:33.746049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.745924 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\") pod \"777f2108-a13f-4a40-b736-41b50534b7c2\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " Oct 30 19:58:33.746049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.745999 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\" (UniqueName: \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\") pod \"777f2108-a13f-4a40-b736-41b50534b7c2\" (UID: \"777f2108-a13f-4a40-b736-41b50534b7c2\") " Oct 30 19:58:33.750158 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.750127 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca" pod "777f2108-a13f-4a40-b736-41b50534b7c2" (UID: "777f2108-a13f-4a40-b736-41b50534b7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:33.750281 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.750127 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca" pod "777f2108-a13f-4a40-b736-41b50534b7c2" (UID: "777f2108-a13f-4a40-b736-41b50534b7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:33.751266 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.751239 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.756136 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.756070 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb" pod "777f2108-a13f-4a40-b736-41b50534b7c2" (UID: "777f2108-a13f-4a40-b736-41b50534b7c2"). InnerVolumeSpecName "kube-api-access-tfhjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:33.865121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.860972 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:33.865121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.861006 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777f2108-a13f-4a40-b736-41b50534b7c2-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:33.865121 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:33.861023 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhjb\" (UniqueName: \"kubernetes.io/projected/777f2108-a13f-4a40-b736-41b50534b7c2-kube-api-access-tfhjb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.019607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.019570 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.024486 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.024452 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:34.028701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.028677 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.028885 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.028863 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:58:34.083634 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.083607 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:58:34.209329 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.209244 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj"] Oct 30 19:58:34.214419 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.214384 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1b36ad-v82pj"] Oct 30 19:58:34.214576 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.214535 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.224742 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.224713 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\" (UniqueName: \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\") pod \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " Oct 30 19:58:34.224945 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.224766 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\") pod \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\" (UID: \"e33c140d-ef70-4b89-91cd-d5bb0078a3a6\") " Oct 30 19:58:34.225941 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.225899 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume" pod "e33c140d-ef70-4b89-91cd-d5bb0078a3a6" (UID: "e33c140d-ef70-4b89-91cd-d5bb0078a3a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.229058 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.229031 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp" pod "e33c140d-ef70-4b89-91cd-d5bb0078a3a6" (UID: "e33c140d-ef70-4b89-91cd-d5bb0078a3a6"). InnerVolumeSpecName "kube-api-access-rxhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.327864 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.327336 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhmp\" (UniqueName: \"kubernetes.io/projected/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-kube-api-access-rxhmp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.328127 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.328086 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e33c140d-ef70-4b89-91cd-d5bb0078a3a6-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.359620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.359580 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.407490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.407458 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a730dd-058b-4a4e-a02c-e03951e1c12b" path="/var/lib/kubelet/pods/f6a730dd-058b-4a4e-a02c-e03951e1c12b/volumes" Oct 30 19:58:34.452607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.452585 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" Oct 30 19:58:34.452750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.452574 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q" event={"ID":"777f2108-a13f-4a40-b736-41b50534b7c2","Type":"ContainerDied","Data":"4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428"} Oct 30 19:58:34.452750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.452682 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5cecb662ab6dc3b746ad3c30c4483bd4c0d4f7a1c2d6b1ca273ad2fc506428" Oct 30 19:58:34.455622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.455592 4990 generic.go:356] "Generic (PLEG): container finished" podID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" containerID="e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d" exitCode=0 Oct 30 19:58:34.455806 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.455781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" event={"ID":"e33c140d-ef70-4b89-91cd-d5bb0078a3a6","Type":"ContainerDied","Data":"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d"} Oct 30 19:58:34.455891 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.455865 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" event={"ID":"e33c140d-ef70-4b89-91cd-d5bb0078a3a6","Type":"ContainerDied","Data":"6c0757a65aed04249ff628ae7666ad79603371226725aa65f909059a661acf05"} Oct 30 19:58:34.455951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.455934 4990 scope.go:122] "RemoveContainer" containerID="e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d" Oct 30 19:58:34.456149 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.456128 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk" Oct 30 19:58:34.485487 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.485403 4990 scope.go:122] "RemoveContainer" containerID="e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d" Oct 30 19:58:34.488009 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:34.487428 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d\": container with ID starting with e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d not found: ID does not exist" containerID="e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d" Oct 30 19:58:34.488320 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.487465 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d"} err="failed to get container status \"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d\": rpc error: code = NotFound desc = could not find container \"e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d\": container with ID starting with e85d8285428cdb6f7e5da00cf6d81c56c6aa160acafd9679bf7209557549b28d not found: ID does not exist" Oct 30 19:58:34.506212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.506188 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:58:34.512505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.512481 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5cc88c857f-vhxdk"] Oct 30 19:58:34.734209 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.734127 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.966892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:34.966850 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:35.097899 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:35.097866 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:58:35.466752 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:35.466659 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:35.467667 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:35.467648 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:58:36.349880 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:36.349844 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.402349 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:36.402319 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" path="/var/lib/kubelet/pods/e33c140d-ef70-4b89-91cd-d5bb0078a3a6/volumes" Oct 30 19:58:36.517618 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:36.517583 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.043171 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:38.043128 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.193242 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:38.193205 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.372117 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:38.367480 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.033105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:39.033065 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk"] Oct 30 19:58:39.042375 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:39.042345 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a885d7db-cvlhk"] Oct 30 19:58:40.154786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:40.154746 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.404393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:40.404358 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0618b6-428f-41d5-8c70-161b161ebfe4" path="/var/lib/kubelet/pods/2b0618b6-428f-41d5-8c70-161b161ebfe4/volumes" Oct 30 19:58:40.749711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:40.749679 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.911018 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:40.910979 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:41.105246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.105122 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:41.598008 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.597967 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp"] Oct 30 19:58:41.599217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599192 4990 cpu_manager.go:414] "containerMap: removing container" podUID="777f2108-a13f-4a40-b736-41b50534b7c2" containerName="revoke-keys" Oct 30 19:58:41.599217 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599217 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f2108-a13f-4a40-b736-41b50534b7c2" containerName="revoke-keys" Oct 30 19:58:41.599362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599256 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" containerName="payload-pre-processing" Oct 30 19:58:41.599362 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599264 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" containerName="payload-pre-processing" Oct 30 19:58:41.599470 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599458 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33c140d-ef70-4b89-91cd-d5bb0078a3a6" containerName="payload-pre-processing" Oct 30 19:58:41.599507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.599485 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f2108-a13f-4a40-b736-41b50534b7c2" containerName="revoke-keys" Oct 30 19:58:41.605645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.605619 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.618304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.618194 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:41.620464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.620442 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp"] Oct 30 19:58:41.623370 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.623345 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:58:41.666072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.666043 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.666235 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.666122 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqz8\" (UniqueName: \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.666388 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.666363 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.767280 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.767240 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.767464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.767443 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.767592 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.767542 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqz8\" (UniqueName: \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.768060 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.768030 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.769951 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.769925 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.784527 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.784500 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqz8\" (UniqueName: \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\") pod \"maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:41.923132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:41.923038 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:42.246291 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:42.235683 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp"] Oct 30 19:58:42.412243 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:42.409901 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.611226 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:42.611183 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" event={"ID":"f13132ed-1155-4a09-bae2-0cf43702c19f","Type":"ContainerStarted","Data":"34dff175792cc962e68e33f17d65e104c27081166a964843a219cdd2a7957b2a"} Oct 30 19:58:42.856995 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:42.856965 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:58:43.048331 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:43.048285 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.186128 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:43.186002 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.658114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:43.657579 4990 generic.go:356] "Generic (PLEG): container finished" podID="f13132ed-1155-4a09-bae2-0cf43702c19f" containerID="35f9924a20691a88b6a96c78c811e382b8d052b4c86ff2e1ee53ee23741a1f1d" exitCode=0 Oct 30 19:58:43.658114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:43.657659 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" event={"ID":"f13132ed-1155-4a09-bae2-0cf43702c19f","Type":"ContainerDied","Data":"35f9924a20691a88b6a96c78c811e382b8d052b4c86ff2e1ee53ee23741a1f1d"} Oct 30 19:58:44.851453 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.851428 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:44.914159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.914124 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:44.921900 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.921794 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\" (UniqueName: \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\") pod \"f13132ed-1155-4a09-bae2-0cf43702c19f\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " Oct 30 19:58:44.922144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.922049 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\") pod \"f13132ed-1155-4a09-bae2-0cf43702c19f\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " Oct 30 19:58:44.922144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.922115 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\") pod \"f13132ed-1155-4a09-bae2-0cf43702c19f\" (UID: \"f13132ed-1155-4a09-bae2-0cf43702c19f\") " Oct 30 19:58:44.923952 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.923889 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca" pod "f13132ed-1155-4a09-bae2-0cf43702c19f" (UID: "f13132ed-1155-4a09-bae2-0cf43702c19f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:44.931770 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.930406 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca" pod "f13132ed-1155-4a09-bae2-0cf43702c19f" (UID: "f13132ed-1155-4a09-bae2-0cf43702c19f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:44.936175 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:44.932298 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8" pod "f13132ed-1155-4a09-bae2-0cf43702c19f" (UID: "f13132ed-1155-4a09-bae2-0cf43702c19f"). InnerVolumeSpecName "kube-api-access-pdqz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:45.024505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.024478 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqz8\" (UniqueName: \"kubernetes.io/projected/f13132ed-1155-4a09-bae2-0cf43702c19f-kube-api-access-pdqz8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:45.024505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.024502 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:45.024505 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.024512 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f13132ed-1155-4a09-bae2-0cf43702c19f-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:45.677255 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.677227 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" Oct 30 19:58:45.677464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.677262 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp" event={"ID":"f13132ed-1155-4a09-bae2-0cf43702c19f","Type":"ContainerDied","Data":"34dff175792cc962e68e33f17d65e104c27081166a964843a219cdd2a7957b2a"} Oct 30 19:58:45.677464 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:45.677320 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34dff175792cc962e68e33f17d65e104c27081166a964843a219cdd2a7957b2a" Oct 30 19:58:46.534838 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:46.532873 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.857893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:47.857855 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" podUID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" containerName="payload-processing" containerID="cri-o://6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086" gracePeriod=25 Oct 30 19:58:48.151761 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.151735 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:58:48.194700 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.194671 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.270250 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.270213 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\" (UniqueName: \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\") pod \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " Oct 30 19:58:48.270432 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.270345 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\") pod \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\" (UID: \"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b\") " Oct 30 19:58:48.271628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.271603 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume" pod "7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" (UID: "7561f0bd-baeb-4c7f-9870-f2e4dd833c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:48.273623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.273595 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m" pod "7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" (UID: "7561f0bd-baeb-4c7f-9870-f2e4dd833c5b"). InnerVolumeSpecName "kube-api-access-46p2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:48.372523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.372442 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p2m\" (UniqueName: \"kubernetes.io/projected/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-kube-api-access-46p2m\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.372523 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.372470 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.709722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.709629 4990 generic.go:356] "Generic (PLEG): container finished" podID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" containerID="6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086" exitCode=0 Oct 30 19:58:48.709722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.709677 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" Oct 30 19:58:48.709966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.709714 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" event={"ID":"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b","Type":"ContainerDied","Data":"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086"} Oct 30 19:58:48.709966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.709772 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6" event={"ID":"7561f0bd-baeb-4c7f-9870-f2e4dd833c5b","Type":"ContainerDied","Data":"bc6c671109a0cbea141a2a8708043a150d82030af6270c8c4896379c4f1246b1"} Oct 30 19:58:48.709966 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.709795 4990 scope.go:122] "RemoveContainer" containerID="6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086" Oct 30 19:58:48.721285 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.721258 4990 scope.go:122] "RemoveContainer" containerID="6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086" Oct 30 19:58:48.721557 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:58:48.721535 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086\": container with ID starting with 6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086 not found: ID does not exist" containerID="6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086" Oct 30 19:58:48.721622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.721563 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086"} err="failed to get container status \"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086\": rpc error: code = NotFound desc = could not find container \"6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086\": container with ID starting with 6ab519d662a52792d629189cc2132764cbd609b778f6980392db706cbf987086 not found: ID does not exist" Oct 30 19:58:48.764201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.764169 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:58:48.770707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:48.770677 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-8598cdcdbb-2v7c6"] Oct 30 19:58:49.029245 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:49.029165 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm"] Oct 30 19:58:49.034736 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:49.034692 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3595fd-8drrm"] Oct 30 19:58:49.925115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:49.924356 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:50.409212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:50.409171 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" path="/var/lib/kubelet/pods/7561f0bd-baeb-4c7f-9870-f2e4dd833c5b/volumes" Oct 30 19:58:50.410103 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:50.410060 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfc2b5d-fd55-4e2f-a2f3-059ec1794199" path="/var/lib/kubelet/pods/fbfc2b5d-fd55-4e2f-a2f3-059ec1794199/volumes" Oct 30 19:58:51.637546 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:51.634115 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.385795 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:53.385758 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.984476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:54.984437 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.327295 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:55.327266 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v_0c4247cd-4e68-4443-8fe8-3f10e59ae5ff/payload-processing/0.log" Oct 30 19:58:56.675782 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.675749 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz"] Oct 30 19:58:56.676493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676476 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" containerName="payload-processing" Oct 30 19:58:56.676493 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676493 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" containerName="payload-processing" Oct 30 19:58:56.676642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676534 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f13132ed-1155-4a09-bae2-0cf43702c19f" containerName="revoke-keys" Oct 30 19:58:56.676642 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676539 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13132ed-1155-4a09-bae2-0cf43702c19f" containerName="revoke-keys" Oct 30 19:58:56.676763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676652 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13132ed-1155-4a09-bae2-0cf43702c19f" containerName="revoke-keys" Oct 30 19:58:56.676763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.676663 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7561f0bd-baeb-4c7f-9870-f2e4dd833c5b" containerName="payload-processing" Oct 30 19:58:56.680692 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.680669 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.694860 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.694732 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:56.704918 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.704890 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:58:56.712641 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.712612 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz"] Oct 30 19:58:56.804696 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.804660 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.804888 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.804798 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdrh\" (UniqueName: \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.804971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.804954 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.906562 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.906532 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.906740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.906668 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.906833 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.906749 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdrh\" (UniqueName: \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.907420 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.907395 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.909176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.909153 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.930711 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.930660 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdrh\" (UniqueName: \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:56.938312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.938292 4990 scope.go:122] "RemoveContainer" containerID="d41553fd0ab7e2b1d1eea30d02cc14ca657bf9beca8cd76f485cb5938ccfe176" Oct 30 19:58:56.949893 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.949877 4990 scope.go:122] "RemoveContainer" containerID="afa31ac9be5d4d2ba96154cd15154471d2c13ea41590205e2645d60ecb3a63a5" Oct 30 19:58:56.997129 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:56.997105 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:58:57.317817 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:57.317788 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz"] Oct 30 19:58:57.319124 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:58:57.319074 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b128b2d_bca1_45e8_aaaa_7608dca8a1fc.slice/crio-b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37 WatchSource:0}: Error finding container b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37: Status 404 returned error can't find the container with id b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37 Oct 30 19:58:57.419704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:57.419672 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:57.832024 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:57.831984 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" event={"ID":"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc","Type":"ContainerStarted","Data":"271f235074ad95c14d99490919d01c80c8a5ae5995b1a5ebce7bd8427a8f08a7"} Oct 30 19:58:57.832476 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:57.832050 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" event={"ID":"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc","Type":"ContainerStarted","Data":"b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37"} Oct 30 19:58:57.877035 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:57.876973 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" podStartSLOduration=1.876955133 podStartE2EDuration="1.876955133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:57.870486009 +0000 UTC m=+3245.582011385" watchObservedRunningTime="2026-10-30 19:58:57.876955133 +0000 UTC m=+3245.588480500" Oct 30 19:58:58.670627 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:58.670592 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.846451 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:58.846409 4990 generic.go:356] "Generic (PLEG): container finished" podID="5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" containerID="271f235074ad95c14d99490919d01c80c8a5ae5995b1a5ebce7bd8427a8f08a7" exitCode=0 Oct 30 19:58:58.846879 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:58.846695 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" event={"ID":"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc","Type":"ContainerDied","Data":"271f235074ad95c14d99490919d01c80c8a5ae5995b1a5ebce7bd8427a8f08a7"} Oct 30 19:58:58.944439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:58.944351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.955504 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:58:59.955467 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.057279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.057257 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:59:00.183274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.183174 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\") pod \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " Oct 30 19:59:00.183274 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.183271 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\" (UniqueName: \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\") pod \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " Oct 30 19:59:00.183530 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.183300 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\") pod \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\" (UID: \"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc\") " Oct 30 19:59:00.183790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.183761 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca" pod "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" (UID: "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:00.184312 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.184286 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:00.185560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.185526 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca" pod "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" (UID: "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:00.185892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.185867 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh" pod "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" (UID: "5b128b2d-bca1-45e8-aaaa-7608dca8a1fc"). InnerVolumeSpecName "kube-api-access-ccdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:00.286185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.286146 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:00.286185 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.286179 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdrh\" (UniqueName: \"kubernetes.io/projected/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc-kube-api-access-ccdrh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:00.867198 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.867153 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" event={"ID":"5b128b2d-bca1-45e8-aaaa-7608dca8a1fc","Type":"ContainerDied","Data":"b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37"} Oct 30 19:59:00.867395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.867167 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz" Oct 30 19:59:00.867395 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:00.867256 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b327326e13f0c5dfd3a46c9c3a523554b224108c0ad583251a492cefae997c37" Oct 30 19:59:02.355623 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:02.355591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.887522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:02.887477 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.515047 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:04.514842 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:05.097131 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:05.097059 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:59:05.700757 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:05.700724 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.080638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:06.080569 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.640287 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:07.640247 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.316808 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:09.316771 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.101750 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:10.101707 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.966309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:10.966272 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.578166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:12.578119 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.173651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:14.173607 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:15.495421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:15.495365 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.036971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.036937 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.120628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.120592 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p"] Oct 30 19:59:16.129425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.129399 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-6e0c55-sxs8p"] Oct 30 19:59:16.191157 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.191086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.338509 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.338475 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.405214 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.405183 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be57a030-1ce2-4853-8c3a-c6f1b4cb2405" path="/var/lib/kubelet/pods/be57a030-1ce2-4853-8c3a-c6f1b4cb2405/volumes" Oct 30 19:59:16.645552 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:16.645457 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.549125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:18.546309 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.088164 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:20.088130 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.229126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:20.229061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.388326 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:20.388250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.860139 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:21.860077 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.599380 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:23.599340 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:24.848672 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:24.848558 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:24.992083 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:24.991949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:25.925109 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:25.925056 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:26.655662 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:26.655611 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.113267 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.113233 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 19:59:27.116975 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.115022 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" containerName="revoke-keys" Oct 30 19:59:27.117201 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.117182 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" containerName="revoke-keys" Oct 30 19:59:27.117551 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.117532 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" containerName="revoke-keys" Oct 30 19:59:27.148002 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.147971 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162547 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162629 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162689 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162715 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162806 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162918 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.162976 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.163067 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j454w\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.163747 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.163143 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.169522 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.167135 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 19:59:27.174928 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.174417 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a76aa2-openshift-default-dockercfg-24mbx\"" Oct 30 19:59:27.190019 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.189977 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.299978 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.299942 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.300740 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.300690 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.301437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.301417 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j454w\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.301844 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.301824 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.302466 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.302448 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.302917 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.302899 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.303400 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.303381 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.303974 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.303940 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.304939 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.304917 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.305561 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.303263 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.305712 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.301305 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.305811 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.302816 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.305901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.304779 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.305994 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.302241 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.308651 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.308609 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.319629 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.319608 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.335903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.335872 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:59:27.336313 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.336281 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" podUID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" containerName="maas-api" containerID="cri-o://1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee" gracePeriod=30 Oct 30 19:59:27.339903 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.339879 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j454w\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.340187 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.340161 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\") pod \"e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.417501 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.417464 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.479322 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.479277 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:27.502768 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.502735 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:59:27.505159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.505123 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" podUID="5af99856-df61-42ee-961e-96d37703979f" containerName="payload-pre-processing" containerID="cri-o://7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c" gracePeriod=30 Oct 30 19:59:27.604831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.604733 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.729105 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.729058 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:59:27.747408 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.747316 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.970421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.963855 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\" (UniqueName: \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\") pod \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " Oct 30 19:59:27.970421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.963961 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\") pod \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " Oct 30 19:59:27.970421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.964061 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\") pod \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " Oct 30 19:59:27.970421 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.964112 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\") pod \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\" (UID: \"d2b4ce16-f26b-4fea-8adb-1826cb1497a5\") " Oct 30 19:59:27.993138 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.993032 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs" pod "d2b4ce16-f26b-4fea-8adb-1826cb1497a5" (UID: "d2b4ce16-f26b-4fea-8adb-1826cb1497a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:27.996790 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.996753 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca" pod "d2b4ce16-f26b-4fea-8adb-1826cb1497a5" (UID: "d2b4ce16-f26b-4fea-8adb-1826cb1497a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:27.997848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:27.997810 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 19:59:28.008191 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.008160 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls" pod "d2b4ce16-f26b-4fea-8adb-1826cb1497a5" (UID: "d2b4ce16-f26b-4fea-8adb-1826cb1497a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:28.009607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.008516 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:28.009607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.008605 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:28.009607 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.008647 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:28.047122 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.047057 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm" pod "d2b4ce16-f26b-4fea-8adb-1826cb1497a5" (UID: "d2b4ce16-f26b-4fea-8adb-1826cb1497a5"). InnerVolumeSpecName "kube-api-access-j2klm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:28.065867 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.065836 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2klm\" (UniqueName: \"kubernetes.io/projected/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-kube-api-access-j2klm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.066013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.065869 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.066013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.065884 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.066013 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.065897 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b4ce16-f26b-4fea-8adb-1826cb1497a5-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.084396 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.084273 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:59:28.170923 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.170885 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\" (UniqueName: \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\") pod \"5af99856-df61-42ee-961e-96d37703979f\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " Oct 30 19:59:28.182154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.182106 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume" pod "5af99856-df61-42ee-961e-96d37703979f" (UID: "5af99856-df61-42ee-961e-96d37703979f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:28.183848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.183630 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:28.187223 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.187199 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l" pod "5af99856-df61-42ee-961e-96d37703979f" (UID: "5af99856-df61-42ee-961e-96d37703979f"). InnerVolumeSpecName "kube-api-access-fzn4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:28.188179 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.188158 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\") pod \"5af99856-df61-42ee-961e-96d37703979f\" (UID: \"5af99856-df61-42ee-961e-96d37703979f\") " Oct 30 19:59:28.188774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.188480 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerStarted","Data":"1afaf109347cfc56597aa7cb458c9d92a79877f637efb5b5808ddb4c3034d2d4"} Oct 30 19:59:28.188774 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.188564 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerStarted","Data":"5a11a1d1c7402670fea1900ca9130cf235b9377beb783f77751ca7284a44e806"} Oct 30 19:59:28.189148 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.189082 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzn4l\" (UniqueName: \"kubernetes.io/projected/5af99856-df61-42ee-961e-96d37703979f-kube-api-access-fzn4l\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.189227 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.189158 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af99856-df61-42ee-961e-96d37703979f-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.190518 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.190497 4990 generic.go:356] "Generic (PLEG): container finished" podID="5af99856-df61-42ee-961e-96d37703979f" containerID="7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c" exitCode=0 Oct 30 19:59:28.190622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.190599 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" event={"ID":"5af99856-df61-42ee-961e-96d37703979f","Type":"ContainerDied","Data":"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c"} Oct 30 19:59:28.190689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.190630 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" event={"ID":"5af99856-df61-42ee-961e-96d37703979f","Type":"ContainerDied","Data":"f7e9cb75028a63f982db8e2f80375464be9acf460492b0b97b9dc277fb3bfb5a"} Oct 30 19:59:28.190689 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.190649 4990 scope.go:122] "RemoveContainer" containerID="7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c" Oct 30 19:59:28.190796 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.190744 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx" Oct 30 19:59:28.198168 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.198142 4990 generic.go:356] "Generic (PLEG): container finished" podID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" containerID="1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee" exitCode=0 Oct 30 19:59:28.198273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.198195 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" Oct 30 19:59:28.198407 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.198391 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" event={"ID":"d2b4ce16-f26b-4fea-8adb-1826cb1497a5","Type":"ContainerDied","Data":"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee"} Oct 30 19:59:28.198467 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.198434 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h" event={"ID":"d2b4ce16-f26b-4fea-8adb-1826cb1497a5","Type":"ContainerDied","Data":"d02c52cc0f70cd07ca97e4cf8924d4997828826eb65038df58eceff61a800e56"} Oct 30 19:59:28.211338 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.211319 4990 scope.go:122] "RemoveContainer" containerID="7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c" Oct 30 19:59:28.211637 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:28.211613 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c\": container with ID starting with 7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c not found: ID does not exist" containerID="7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c" Oct 30 19:59:28.211699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.211648 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c"} err="failed to get container status \"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c\": rpc error: code = NotFound desc = could not find container \"7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c\": container with ID starting with 7fd5471993285eb8430cb6ed06f8e6bef077bd4672dc5dc6ab24a4183381854c not found: ID does not exist" Oct 30 19:59:28.211699 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.211672 4990 scope.go:122] "RemoveContainer" containerID="1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee" Oct 30 19:59:28.226200 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.226179 4990 scope.go:122] "RemoveContainer" containerID="1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee" Oct 30 19:59:28.226462 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:28.226444 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee\": container with ID starting with 1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee not found: ID does not exist" containerID="1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee" Oct 30 19:59:28.226533 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.226472 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee"} err="failed to get container status \"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee\": rpc error: code = NotFound desc = could not find container \"1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee\": container with ID starting with 1a18e48665086646990d568cab46896e97047a9ee7c01608a163b64ce29b0fee not found: ID does not exist" Oct 30 19:59:28.238166 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.238122 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podStartSLOduration=1.238109422 podStartE2EDuration="1.238109422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:28.232859591 +0000 UTC m=+3275.944384957" watchObservedRunningTime="2026-10-30 19:59:28.238109422 +0000 UTC m=+3275.949634795" Oct 30 19:59:28.263162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.263139 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp"] Oct 30 19:59:28.268819 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.268788 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59f9ed-bd8tp"] Oct 30 19:59:28.273831 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.273785 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:59:28.278353 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.278328 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-889559b99-24qrx"] Oct 30 19:59:28.308022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.307993 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:59:28.314364 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.314339 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-59f9ed-7c7d9f5dd-5x75h"] Oct 30 19:59:28.340430 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.340393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:28.408766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.408730 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af99856-df61-42ee-961e-96d37703979f" path="/var/lib/kubelet/pods/5af99856-df61-42ee-961e-96d37703979f/volumes" Oct 30 19:59:28.409284 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.409263 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" path="/var/lib/kubelet/pods/d2b4ce16-f26b-4fea-8adb-1826cb1497a5/volumes" Oct 30 19:59:28.409766 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.409749 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13132ed-1155-4a09-bae2-0cf43702c19f" path="/var/lib/kubelet/pods/f13132ed-1155-4a09-bae2-0cf43702c19f/volumes" Oct 30 19:59:28.480162 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.480057 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:28.787439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:28.787353 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.488554 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:29.488521 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:30.190114 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:30.188740 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:30.221304 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:30.221233 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:30.222439 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:30.222413 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 19:59:31.768072 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:31.768036 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:32.909745 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:32.909710 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.072393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:33.072343 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.212238 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:33.212144 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:59:33.212444 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:33.212412 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" containerID="cri-o://fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d" gracePeriod=30 Oct 30 19:59:33.325707 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:33.325666 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.556025 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:33.555985 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.367510 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:34.367470 4990 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:59:34.368001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:34.367536 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:59:35.100231 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:35.100190 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 19:59:35.253925 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:35.253883 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.427032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:35.426945 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.589180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:35.589137 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.019809 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.019777 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9"] Oct 30 19:59:36.020516 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020501 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" containerName="maas-api" Oct 30 19:59:36.020571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020517 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" containerName="maas-api" Oct 30 19:59:36.020571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020537 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5af99856-df61-42ee-961e-96d37703979f" containerName="payload-pre-processing" Oct 30 19:59:36.020571 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020544 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af99856-df61-42ee-961e-96d37703979f" containerName="payload-pre-processing" Oct 30 19:59:36.020674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020657 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b4ce16-f26b-4fea-8adb-1826cb1497a5" containerName="maas-api" Oct 30 19:59:36.020674 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.020669 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af99856-df61-42ee-961e-96d37703979f" containerName="payload-pre-processing" Oct 30 19:59:36.025999 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.025983 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.042606 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.042581 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.042777 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.042757 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.042826 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.042792 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57sl\" (UniqueName: \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.047722 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.047703 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:36.052620 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.052592 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:59:36.058225 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.058199 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9"] Oct 30 19:59:36.144366 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.144335 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.144521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.144461 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.144521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.144483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57sl\" (UniqueName: \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.144985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.144961 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.146892 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.146875 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.165875 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.165855 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57sl\" (UniqueName: \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.340442 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.340410 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:36.695738 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:36.684529 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9"] Oct 30 19:59:37.287111 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:37.279054 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:37.304987 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:37.304950 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" event={"ID":"ed008c61-7716-42f4-a131-a45aa525da2f","Type":"ContainerStarted","Data":"f89216ba228c0743d1e23ecc94c91aff5809831415c13476296fd141312965f0"} Oct 30 19:59:38.315154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.315125 4990 generic.go:356] "Generic (PLEG): container finished" podID="ed008c61-7716-42f4-a131-a45aa525da2f" containerID="321440e2e69fa7a1723c6ca9d5c7b02ef7ec70b5c5d0bf08855dd9b9058b5814" exitCode=0 Oct 30 19:59:38.315491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.315211 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" event={"ID":"ed008c61-7716-42f4-a131-a45aa525da2f","Type":"ContainerDied","Data":"321440e2e69fa7a1723c6ca9d5c7b02ef7ec70b5c5d0bf08855dd9b9058b5814"} Oct 30 19:59:38.538477 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.538448 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592131 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592216 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592263 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592340 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592399 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592451 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592486 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592622 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.592698 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\") pod \"efc5ee35-7072-498a-b6ca-f4b82af26f06\" (UID: \"efc5ee35-7072-498a-b6ca-f4b82af26f06\") " Oct 30 19:59:38.593694 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.593428 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:38.595086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.594565 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:38.595086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.594755 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:38.595086 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.595048 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:38.612622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.610347 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:38.612622 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.612284 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:38.614219 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.614170 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:38.615489 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.615445 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:38.623178 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.621613 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g" pod "efc5ee35-7072-498a-b6ca-f4b82af26f06" (UID: "efc5ee35-7072-498a-b6ca-f4b82af26f06"). InnerVolumeSpecName "kube-api-access-d894g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:38.699739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699671 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.699739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699706 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc5ee35-7072-498a-b6ca-f4b82af26f06-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.699739 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699721 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699748 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699762 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699776 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699789 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699800 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc5ee35-7072-498a-b6ca-f4b82af26f06-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:38.700037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:38.699812 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d894g\" (UniqueName: \"kubernetes.io/projected/efc5ee35-7072-498a-b6ca-f4b82af26f06-kube-api-access-d894g\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:39.047570 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.047534 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j"] Oct 30 19:59:39.050015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.049967 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:39.053943 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.053903 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fabdda-67q9j"] Oct 30 19:59:39.325628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.325593 4990 generic.go:356] "Generic (PLEG): container finished" podID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerID="fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d" exitCode=0 Oct 30 19:59:39.326043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.325678 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" Oct 30 19:59:39.326043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.325670 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerDied","Data":"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d"} Oct 30 19:59:39.326043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.325839 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz" event={"ID":"efc5ee35-7072-498a-b6ca-f4b82af26f06","Type":"ContainerDied","Data":"6f7c3f82f592b8ec40459fb8d84c7bbafdad83d5ce61ba2d3cccbe590a341065"} Oct 30 19:59:39.326043 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.325874 4990 scope.go:122] "RemoveContainer" containerID="fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d" Oct 30 19:59:39.339786 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.339765 4990 scope.go:122] "RemoveContainer" containerID="dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a" Oct 30 19:59:39.353906 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.353885 4990 scope.go:122] "RemoveContainer" containerID="fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d" Oct 30 19:59:39.354221 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:39.354193 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d\": container with ID starting with fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d not found: ID does not exist" containerID="fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d" Oct 30 19:59:39.354354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.354228 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d"} err="failed to get container status \"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d\": rpc error: code = NotFound desc = could not find container \"fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d\": container with ID starting with fdad0f1e63ae30471423ba2dae43a7fd3e8749bda8e46f5d9d7b7fc8a99edf4d not found: ID does not exist" Oct 30 19:59:39.354354 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.354250 4990 scope.go:122] "RemoveContainer" containerID="dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a" Oct 30 19:59:39.354544 ip-10-0-135-68 kubenswrapper[4990]: E1030 19:59:39.354525 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a\": container with ID starting with dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a not found: ID does not exist" containerID="dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a" Oct 30 19:59:39.354590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.354552 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a"} err="failed to get container status \"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a\": rpc error: code = NotFound desc = could not find container \"dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a\": container with ID starting with dd8d446738dd1fe1a5d3e6a0f1091f6bd799a62a6cf8c45c580ac3816c48ca2a not found: ID does not exist" Oct 30 19:59:39.397468 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.397434 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:59:39.405001 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.404972 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-kmxvz"] Oct 30 19:59:39.553525 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.553494 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:39.615862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.613424 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\") pod \"ed008c61-7716-42f4-a131-a45aa525da2f\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " Oct 30 19:59:39.615862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.613606 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\" (UniqueName: \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\") pod \"ed008c61-7716-42f4-a131-a45aa525da2f\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " Oct 30 19:59:39.615862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.613647 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\") pod \"ed008c61-7716-42f4-a131-a45aa525da2f\" (UID: \"ed008c61-7716-42f4-a131-a45aa525da2f\") " Oct 30 19:59:39.615862 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.614333 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca" pod "ed008c61-7716-42f4-a131-a45aa525da2f" (UID: "ed008c61-7716-42f4-a131-a45aa525da2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:39.617125 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.617083 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl" pod "ed008c61-7716-42f4-a131-a45aa525da2f" (UID: "ed008c61-7716-42f4-a131-a45aa525da2f"). InnerVolumeSpecName "kube-api-access-w57sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:39.617507 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.617483 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca" pod "ed008c61-7716-42f4-a131-a45aa525da2f" (UID: "ed008c61-7716-42f4-a131-a45aa525da2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:39.715474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.715440 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57sl\" (UniqueName: \"kubernetes.io/projected/ed008c61-7716-42f4-a131-a45aa525da2f-kube-api-access-w57sl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:39.715474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.715466 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:39.715474 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:39.715478 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed008c61-7716-42f4-a131-a45aa525da2f-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:40.336463 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.336427 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" Oct 30 19:59:40.336984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.336463 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9" event={"ID":"ed008c61-7716-42f4-a131-a45aa525da2f","Type":"ContainerDied","Data":"f89216ba228c0743d1e23ecc94c91aff5809831415c13476296fd141312965f0"} Oct 30 19:59:40.336984 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.336519 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89216ba228c0743d1e23ecc94c91aff5809831415c13476296fd141312965f0" Oct 30 19:59:40.403628 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.403594 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7855042-e67b-441c-b7b9-49e47aea7015" path="/var/lib/kubelet/pods/b7855042-e67b-441c-b7b9-49e47aea7015/volumes" Oct 30 19:59:40.404169 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.404148 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" path="/var/lib/kubelet/pods/efc5ee35-7072-498a-b6ca-f4b82af26f06/volumes" Oct 30 19:59:40.549781 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.549751 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb"] Oct 30 19:59:40.550485 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550470 4990 cpu_manager.go:414] "containerMap: removing container" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550486 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550506 4990 cpu_manager.go:414] "containerMap: removing container" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550511 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550528 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ed008c61-7716-42f4-a131-a45aa525da2f" containerName="revoke-keys" Oct 30 19:59:40.550539 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550534 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed008c61-7716-42f4-a131-a45aa525da2f" containerName="revoke-keys" Oct 30 19:59:40.550730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550630 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550640 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc5ee35-7072-498a-b6ca-f4b82af26f06" containerName="istio-proxy" Oct 30 19:59:40.550730 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.550648 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed008c61-7716-42f4-a131-a45aa525da2f" containerName="revoke-keys" Oct 30 19:59:40.555479 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.555460 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.593167 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.592767 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:40.597113 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.594183 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:59:40.634457 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.634394 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb"] Oct 30 19:59:40.639273 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.639239 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.639482 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.639462 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjchq\" (UniqueName: \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.639577 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.639504 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.740585 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.740553 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.740843 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.740797 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjchq\" (UniqueName: \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.740992 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.740864 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.741361 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.741342 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.743049 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.743027 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.762132 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.759005 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjchq\" (UniqueName: \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:40.939520 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:40.938916 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:41.300491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:41.300354 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb"] Oct 30 19:59:41.303877 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:59:41.303688 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe45fab4_338c_41e6_9515_1168d477466c.slice/crio-7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51 WatchSource:0}: Error finding container 7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51: Status 404 returned error can't find the container with id 7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51 Oct 30 19:59:41.352137 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:41.352111 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" event={"ID":"fe45fab4-338c-41e6-9515-1168d477466c","Type":"ContainerStarted","Data":"7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51"} Oct 30 19:59:41.447962 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:41.447868 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.422645 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:42.422608 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" event={"ID":"fe45fab4-338c-41e6-9515-1168d477466c","Type":"ContainerStarted","Data":"61c1942ab63b44fb4e10fddff1ebdb7b313baeef9b79b0f88a0a06912f1c42cf"} Oct 30 19:59:42.461409 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:42.461355 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" podStartSLOduration=2.461341159 podStartE2EDuration="2.461341159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:42.456575216 +0000 UTC m=+3290.168100583" watchObservedRunningTime="2026-10-30 19:59:42.461341159 +0000 UTC m=+3290.172866525" Oct 30 19:59:43.415664 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.415629 4990 generic.go:356] "Generic (PLEG): container finished" podID="fe45fab4-338c-41e6-9515-1168d477466c" containerID="61c1942ab63b44fb4e10fddff1ebdb7b313baeef9b79b0f88a0a06912f1c42cf" exitCode=0 Oct 30 19:59:43.415837 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.415709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" event={"ID":"fe45fab4-338c-41e6-9515-1168d477466c","Type":"ContainerDied","Data":"61c1942ab63b44fb4e10fddff1ebdb7b313baeef9b79b0f88a0a06912f1c42cf"} Oct 30 19:59:43.494602 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.494568 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9"] Oct 30 19:59:43.499725 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.499696 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.503534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.503508 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9"] Oct 30 19:59:43.611015 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.610983 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ms49\" (UniqueName: \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.611206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.611023 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.611206 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.611196 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.712022 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.711935 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.712203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.712072 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ms49\" (UniqueName: \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.712203 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.712130 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.712915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.712891 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.714425 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.714401 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.735300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.735277 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ms49\" (UniqueName: \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\") pod \"maas-api-revoke-sub-db458b49-fnsr9\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.828147 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.828112 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:43.930365 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:43.930119 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.210275 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.210248 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9"] Oct 30 19:59:44.213442 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:59:44.213414 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbe93e8_fb40_4fea_bce0_fbefb143b0ff.slice/crio-a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3 WatchSource:0}: Error finding container a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3: Status 404 returned error can't find the container with id a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3 Oct 30 19:59:44.425437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.425401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" event={"ID":"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff","Type":"ContainerStarted","Data":"a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3"} Oct 30 19:59:44.610300 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.610272 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:44.731212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.731181 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\") pod \"fe45fab4-338c-41e6-9515-1168d477466c\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " Oct 30 19:59:44.731324 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.731306 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\" (UniqueName: \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\") pod \"fe45fab4-338c-41e6-9515-1168d477466c\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " Oct 30 19:59:44.731506 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.731486 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\") pod \"fe45fab4-338c-41e6-9515-1168d477466c\" (UID: \"fe45fab4-338c-41e6-9515-1168d477466c\") " Oct 30 19:59:44.731829 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.731806 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca" pod "fe45fab4-338c-41e6-9515-1168d477466c" (UID: "fe45fab4-338c-41e6-9515-1168d477466c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:44.732152 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.732128 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:44.733307 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.733286 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca" pod "fe45fab4-338c-41e6-9515-1168d477466c" (UID: "fe45fab4-338c-41e6-9515-1168d477466c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:44.734899 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.734878 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq" pod "fe45fab4-338c-41e6-9515-1168d477466c" (UID: "fe45fab4-338c-41e6-9515-1168d477466c"). InnerVolumeSpecName "kube-api-access-kjchq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:44.833475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.833441 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe45fab4-338c-41e6-9515-1168d477466c-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:44.833475 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:44.833478 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjchq\" (UniqueName: \"kubernetes.io/projected/fe45fab4-338c-41e6-9515-1168d477466c-kube-api-access-kjchq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:45.437033 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:45.436999 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" event={"ID":"fe45fab4-338c-41e6-9515-1168d477466c","Type":"ContainerDied","Data":"7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51"} Oct 30 19:59:45.437276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:45.437058 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7961d5eb62c98874e1f0a2e9975252bf60c5d3de8d7acd112ea77a044d603d51" Oct 30 19:59:45.437276 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:45.437007 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb" Oct 30 19:59:45.445293 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:45.445264 4990 generic.go:356] "Generic (PLEG): container finished" podID="ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" containerID="6a4b9f44030ac8f952b793fc8fe6d392b777631e43d838e2bb5e0ec0da803e1d" exitCode=0 Oct 30 19:59:45.445437 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:45.445307 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" event={"ID":"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff","Type":"ContainerDied","Data":"6a4b9f44030ac8f952b793fc8fe6d392b777631e43d838e2bb5e0ec0da803e1d"} Oct 30 19:59:46.022704 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.021895 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.555845 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.555810 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 19:59:46.605283 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.605070 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 19:59:46.671529 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.671488 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:46.768355 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.768267 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\") pod \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " Oct 30 19:59:46.768534 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.768471 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\") pod \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " Oct 30 19:59:46.768648 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.768616 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\" (UniqueName: \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\") pod \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\" (UID: \"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff\") " Oct 30 19:59:46.772360 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.772317 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca" pod "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" (UID: "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:46.772852 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.772825 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49" pod "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" (UID: "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff"). InnerVolumeSpecName "kube-api-access-4ms49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:46.785176 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.775736 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca" pod "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" (UID: "ebbe93e8-fb40-4fea-bce0-fbefb143b0ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:46.872632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.872593 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ms49\" (UniqueName: \"kubernetes.io/projected/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-kube-api-access-4ms49\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:46.872632 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.872631 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:46.872848 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:46.872646 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 19:59:47.438846 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:47.438810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:47.467309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:47.467280 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" Oct 30 19:59:47.467309 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:47.467299 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9" event={"ID":"ebbe93e8-fb40-4fea-bce0-fbefb143b0ff","Type":"ContainerDied","Data":"a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3"} Oct 30 19:59:47.467490 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:47.467357 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a320eae41a842e7edc2feb0f7be51a8394030c6ecb25145cd40980aecd4341d3" Oct 30 19:59:47.832948 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:47.832801 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:48.481374 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.481303 4990 generic.go:356] "Generic (PLEG): container finished" podID="446334fe-4d24-4e9a-9968-b02208c0a647" containerID="2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d" exitCode=0 Oct 30 19:59:48.481854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.481377 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerDied","Data":"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d"} Oct 30 19:59:48.481854 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.481784 4990 scope.go:122] "RemoveContainer" containerID="2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d" Oct 30 19:59:48.483564 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.483535 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:48.483650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.483604 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:48.483650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:48.483637 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 19:59:49.393701 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.393653 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.496135 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.496067 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerStarted","Data":"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a"} Oct 30 19:59:49.551579 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.549753 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.709726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.709631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.959590 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.959556 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:59:49.960832 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.960757 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:49.960971 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:49.960828 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:50.030763 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.030723 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw"] Oct 30 19:59:50.035190 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.035164 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f7efe73-qlkkw"] Oct 30 19:59:50.409032 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.408993 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bccd00-d3d4-427e-9521-fb178021daa6" path="/var/lib/kubelet/pods/b7bccd00-d3d4-427e-9521-fb178021daa6/volumes" Oct 30 19:59:50.475115 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.470172 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:59:50.959246 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.959209 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:50.959644 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:50.959256 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:51.197650 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.197597 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.306882 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.306840 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 19:59:51.307715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307691 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fe45fab4-338c-41e6-9515-1168d477466c" containerName="revoke-keys" Oct 30 19:59:51.307715 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307714 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe45fab4-338c-41e6-9515-1168d477466c" containerName="revoke-keys" Oct 30 19:59:51.307919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307770 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" containerName="revoke-keys" Oct 30 19:59:51.307919 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307778 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" containerName="revoke-keys" Oct 30 19:59:51.308037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307945 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" containerName="revoke-keys" Oct 30 19:59:51.308037 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.307970 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe45fab4-338c-41e6-9515-1168d477466c" containerName="revoke-keys" Oct 30 19:59:51.312279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.312259 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.326212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.326190 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a76aa2\"" Oct 30 19:59:51.332079 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.332057 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 19:59:51.336231 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.336211 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a76aa2-dockercfg-2b2qf\"" Oct 30 19:59:51.375126 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.361588 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.441124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.423057 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.441124 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.423201 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf72x\" (UniqueName: \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.593212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.580489 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" containerID="cri-o://1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791" gracePeriod=25 Oct 30 19:59:51.593212 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.581679 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 19:59:51.661144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.597513 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.661144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.597641 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf72x\" (UniqueName: \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.661144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.620447 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" containerID="cri-o://cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856" gracePeriod=25 Oct 30 19:59:51.661144 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.646477 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.675112 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.667103 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.712154 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.712067 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.712381 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.712358 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k27n\" (UniqueName: \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.717904 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.717850 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.718901 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.718873 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf72x\" (UniqueName: \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.734159 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.727348 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 19:59:51.814521 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.814480 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k27n\" (UniqueName: \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.814726 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.814689 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.815614 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.815587 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.832985 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.832956 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k27n\" (UniqueName: \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\") pod \"payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:51.926944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.926871 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:51.960292 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.960262 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:51.960638 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:51.960332 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:52.052560 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.052524 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:52.271448 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.271400 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 19:59:52.307662 ip-10-0-135-68 kubenswrapper[4990]: W1030 19:59:52.307588 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b82cbfb_4630_44b0_83a9_a38a5d48ff73.slice/crio-c0653799fe08da59449e233ba2b1130ce8421e086c41dfddbfded57b8bf8ca97 WatchSource:0}: Error finding container c0653799fe08da59449e233ba2b1130ce8421e086c41dfddbfded57b8bf8ca97: Status 404 returned error can't find the container with id c0653799fe08da59449e233ba2b1130ce8421e086c41dfddbfded57b8bf8ca97 Oct 30 19:59:52.422612 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.422576 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 19:59:52.759036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.758947 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" event={"ID":"4b82cbfb-4630-44b0-83a9-a38a5d48ff73","Type":"ContainerStarted","Data":"5026e2e426fcb9c088cacc659687d28ecd3343651ab653e5744b6b5c159ca61d"} Oct 30 19:59:52.759036 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.759014 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" event={"ID":"4b82cbfb-4630-44b0-83a9-a38a5d48ff73","Type":"ContainerStarted","Data":"c0653799fe08da59449e233ba2b1130ce8421e086c41dfddbfded57b8bf8ca97"} Oct 30 19:59:52.759279 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.759044 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 19:59:52.761234 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.761200 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" event={"ID":"8e8bf6bf-b063-416d-923b-bcf21e98f692","Type":"ContainerStarted","Data":"534ce59bcf065bda5ac524b47030a50542c88bce4c1cea67fbeffd86f798276b"} Oct 30 19:59:52.761393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.761256 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" event={"ID":"8e8bf6bf-b063-416d-923b-bcf21e98f692","Type":"ContainerStarted","Data":"7864f3328e959310b687e60e1059a59082d6e32c6e41796696b37ca372c2cc96"} Oct 30 19:59:52.761393 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.761312 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 19:59:52.813298 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.813248 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" podStartSLOduration=1.81323443 podStartE2EDuration="1.81323443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:52.810441416 +0000 UTC m=+3300.521966782" watchObservedRunningTime="2026-10-30 19:59:52.81323443 +0000 UTC m=+3300.524759796" Oct 30 19:59:52.857180 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.857132 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" podStartSLOduration=1.857119273 podStartE2EDuration="1.857119273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:52.852298912 +0000 UTC m=+3300.563824279" watchObservedRunningTime="2026-10-30 19:59:52.857119273 +0000 UTC m=+3300.568644641" Oct 30 19:59:52.959402 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.959369 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:52.959572 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:52.959421 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:53.415816 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:53.413811 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:53.959247 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:53.959211 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:53.959433 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:53.959267 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:54.226039 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.225955 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:54.647915 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.647882 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:59:54.657177 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.657152 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:59:54.702428 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.702403 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 19:59:54.713695 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.713670 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 19:59:54.959944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.959858 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body= Oct 30 19:59:54.959944 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:54.959912 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" Oct 30 19:59:55.016119 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:55.014502 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.235491 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:55.235404 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:55.964583 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:55.964553 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:59:55.965916 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:55.965886 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 19:59:56.828688 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:56.828607 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:57.052084 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:57.052055 4990 scope.go:122] "RemoveContainer" containerID="43be77d766b9d26dfce8f6034f1bfa402d715540f00b4eeda1a5507fa3238efd" Oct 30 19:59:57.063438 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:57.063419 4990 scope.go:122] "RemoveContainer" containerID="3e399028075b49338d93a4b754a77d462440ea97b027cc638052be07535141a5" Oct 30 19:59:57.074566 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:57.074543 4990 scope.go:122] "RemoveContainer" containerID="161ea29a5413ecd0af65bf1b6ef99b9e54eecdbd5649e0fe23f39c1385429acd" Oct 30 19:59:58.497677 ip-10-0-135-68 kubenswrapper[4990]: I1030 19:59:58.497047 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.145211 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.145178 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:00.154202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.154175 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.154634 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.154605 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:00.160181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.160160 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.162766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.162741 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:00.168352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.168333 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.170913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.170894 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:00:00.176405 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.176381 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:00.181285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.181262 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.190603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.190573 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:00.199934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.197170 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:00.204162 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.203696 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:00.210443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.209911 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:00.210443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.210020 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.217924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.217697 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:00.230944 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.230181 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.241999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.241976 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:00.248961 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.248594 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:00.263586 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.259946 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:00.287113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.278470 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297190 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297308 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jc4\" (UniqueName: \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297396 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297574 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297655 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297710 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfn6f\" (UniqueName: \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297747 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tksm\" (UniqueName: \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.302112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.297792 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf85q\" (UniqueName: \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.330114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.323624 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:00.334110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.333345 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.362076 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.362041 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:00.366978 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.366947 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:00.371395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.371371 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:00.377166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.377146 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.378038 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.378015 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:00.384364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.384085 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.391535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.391515 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:00.396642 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.396275 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.399579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399551 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.399666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399592 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.399666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399656 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.399783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399711 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfn6f\" (UniqueName: \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.399851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399776 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tksm\" (UniqueName: \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.399851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399828 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf85q\" (UniqueName: \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.399965 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399876 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdrw\" (UniqueName: \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.399965 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399936 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cbc\" (UniqueName: \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.400072 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.399969 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.400072 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400010 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.400228 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400132 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jc4\" (UniqueName: \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.400228 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400175 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.400342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400253 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.400434 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400415 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6hf\" (UniqueName: \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.400579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400554 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.400806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.400781 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.401408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.401385 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.401530 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.401507 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.408115 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.408075 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:00.408208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.408138 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:00.408208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.408157 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:00.411724 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.411706 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:00.420632 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.420609 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jc4\" (UniqueName: \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\") pod \"maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.423408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.421776 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf85q\" (UniqueName: \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.423408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.421834 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tksm\" (UniqueName: \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\") pod \"maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.423408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.423298 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfn6f\" (UniqueName: \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.479030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.479000 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:00.487340 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.487312 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:00.499717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.499695 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:00.501534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501508 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdrw\" (UniqueName: \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.501641 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501561 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv79\" (UniqueName: \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.501709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501679 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cbc\" (UniqueName: \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.501772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501731 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4qw\" (UniqueName: \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.501834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501778 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.501930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501907 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.501975 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzzm\" (UniqueName: \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.502114 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnn6\" (UniqueName: \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.502146 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.502187 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.502216 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6hf\" (UniqueName: \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.502683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.502277 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.503773 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.503744 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.503962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.503944 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.504047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.504026 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.504134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.504077 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.504643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.504620 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.522935 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.522911 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6hf\" (UniqueName: \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.528695 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.528671 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdrw\" (UniqueName: \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.530741 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.530719 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cbc\" (UniqueName: \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.542033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.541995 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:00.560665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.560331 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:00.602585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.602378 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608202 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzzm\" (UniqueName: \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608315 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnn6\" (UniqueName: \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608345 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608380 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608436 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608469 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.611912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.608582 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv79\" (UniqueName: \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.615197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.614937 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.615197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.611070 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.615197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.613738 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.615197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.614939 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4qw\" (UniqueName: \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.615197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.611005 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.624534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.624388 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:00.632268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.632229 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4qw\" (UniqueName: \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.634630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.632575 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnn6\" (UniqueName: \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\") pod \"maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.634630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.632761 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzzm\" (UniqueName: \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\") pod \"maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.642108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.642051 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv79\" (UniqueName: \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\") pod \"maas-api-key-cleanup-29889840-mx55h\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.655002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.654520 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:00.708137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.701369 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:00.708137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.706346 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:00.772386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.771945 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:00.793156 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.781705 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.825530 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.824506 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:00.840989 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:00.840943 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62372dcf_ea83_4c7a_a3f9_96007c3aafcb.slice/crio-8cd17bbaa57e5838f32339d9502615f649958f8224ecca13d54e6d18daa6a1a3 WatchSource:0}: Error finding container 8cd17bbaa57e5838f32339d9502615f649958f8224ecca13d54e6d18daa6a1a3: Status 404 returned error can't find the container with id 8cd17bbaa57e5838f32339d9502615f649958f8224ecca13d54e6d18daa6a1a3 Oct 30 20:00:00.958927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:00.958885 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerStarted","Data":"8cd17bbaa57e5838f32339d9502615f649958f8224ecca13d54e6d18daa6a1a3"} Oct 30 20:00:01.034289 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.034224 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw"] Oct 30 20:00:01.039836 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.039804 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:01.044352 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.044313 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467b4713_0df1_4d0e_82bb_222aa22f7936.slice/crio-0c8eed1f3c00236d2be4d801c08d6e551821a1b9dca5526a7ad3ac23c7e0aed2 WatchSource:0}: Error finding container 0c8eed1f3c00236d2be4d801c08d6e551821a1b9dca5526a7ad3ac23c7e0aed2: Status 404 returned error can't find the container with id 0c8eed1f3c00236d2be4d801c08d6e551821a1b9dca5526a7ad3ac23c7e0aed2 Oct 30 20:00:01.045610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.045555 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb8362b-29tmw"] Oct 30 20:00:01.066200 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.066126 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:01.214471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.214038 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:01.217918 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.217882 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d9402f_122c_4206_a6e9_0ceda82c6117.slice/crio-a0b82453ee8e924bbc50ed759f323ac5669708603c68baee10dac2e06fb6ba13 WatchSource:0}: Error finding container a0b82453ee8e924bbc50ed759f323ac5669708603c68baee10dac2e06fb6ba13: Status 404 returned error can't find the container with id a0b82453ee8e924bbc50ed759f323ac5669708603c68baee10dac2e06fb6ba13 Oct 30 20:00:01.237919 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.237890 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:01.246398 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.246374 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c70006_7ab0_4e77_bd20_3689b18ebdb5.slice/crio-d6dc97d44a576d7133372b7ccf422b62e09eca4386e4746dbb42da96f02fb554 WatchSource:0}: Error finding container d6dc97d44a576d7133372b7ccf422b62e09eca4386e4746dbb42da96f02fb554: Status 404 returned error can't find the container with id d6dc97d44a576d7133372b7ccf422b62e09eca4386e4746dbb42da96f02fb554 Oct 30 20:00:01.305482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.305457 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:01.352315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.352294 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:01.405777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.405743 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:01.410627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.410601 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:01.436883 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.436859 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:01.466471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.466447 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:01.470311 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.470281 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596026b3_cdd7_4b4e_8416_d02c7b78364e.slice/crio-51598ae37846e1d1165ce713358f5fb4a008f64685af4512091bde908e4a5e8e WatchSource:0}: Error finding container 51598ae37846e1d1165ce713358f5fb4a008f64685af4512091bde908e4a5e8e: Status 404 returned error can't find the container with id 51598ae37846e1d1165ce713358f5fb4a008f64685af4512091bde908e4a5e8e Oct 30 20:00:01.473052 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.473029 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782200a6_b9fd_491c_88e5_c0af7ab845d1.slice/crio-0f467157c6d253f105c0341a33566e665daea927ac8e06f761a2e1dab7f5e105 WatchSource:0}: Error finding container 0f467157c6d253f105c0341a33566e665daea927ac8e06f761a2e1dab7f5e105: Status 404 returned error can't find the container with id 0f467157c6d253f105c0341a33566e665daea927ac8e06f761a2e1dab7f5e105 Oct 30 20:00:01.474837 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:01.474818 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cb01c5_4e7f_4cd5_8bd4_08c3148341cc.slice/crio-4152dd36506f89e0083eab5ecd6b81a081fabd9ede4bdebb873aafa32327c911 WatchSource:0}: Error finding container 4152dd36506f89e0083eab5ecd6b81a081fabd9ede4bdebb873aafa32327c911: Status 404 returned error can't find the container with id 4152dd36506f89e0083eab5ecd6b81a081fabd9ede4bdebb873aafa32327c911 Oct 30 20:00:01.972833 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.972728 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerStarted","Data":"4152dd36506f89e0083eab5ecd6b81a081fabd9ede4bdebb873aafa32327c911"} Oct 30 20:00:01.974746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.974712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerStarted","Data":"0f467157c6d253f105c0341a33566e665daea927ac8e06f761a2e1dab7f5e105"} Oct 30 20:00:01.977028 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.976998 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerStarted","Data":"51598ae37846e1d1165ce713358f5fb4a008f64685af4512091bde908e4a5e8e"} Oct 30 20:00:01.980891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.980866 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" event={"ID":"c0c70006-7ab0-4e77-bd20-3689b18ebdb5","Type":"ContainerStarted","Data":"d6dc97d44a576d7133372b7ccf422b62e09eca4386e4746dbb42da96f02fb554"} Oct 30 20:00:01.982737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.982713 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerStarted","Data":"9cd270a8f77b50f9ac7b2333d0aa0968dd8b37b7ab238439deac94bb10015ea0"} Oct 30 20:00:01.982864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.982759 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerStarted","Data":"a0b82453ee8e924bbc50ed759f323ac5669708603c68baee10dac2e06fb6ba13"} Oct 30 20:00:01.984223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.984201 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" event={"ID":"914c666a-e17b-480d-8512-a90e7e8c165c","Type":"ContainerStarted","Data":"111383d07b81cb3d833d6b0b88dd7aab2f3b4a95b3f83622b43b04d0b78da55b"} Oct 30 20:00:01.985521 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.985500 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerStarted","Data":"a1f01c1e7470d498016f8a92f7268927c3b9ec36a9232ebcc4a08f05d0dfd2f7"} Oct 30 20:00:01.987117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.987082 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerStarted","Data":"5a61ef1c68290bb17c1a33707ad93df83406d9fd6ff5b573897db36b82479661"} Oct 30 20:00:01.988406 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.988387 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerStarted","Data":"d838b3ab55ea9b4c4b27173f2e5305e1da7c48fa720ac382ad37158769777e9e"} Oct 30 20:00:01.989858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.989835 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerStarted","Data":"da240b11c1b198b59bf1fba3061f46f697ff5b74f7f661b0c723107ce7605e8c"} Oct 30 20:00:01.989961 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.989883 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerStarted","Data":"3de51ad214092a3806315b5f0ebef3a00d7c1b103dbaedb6ef852ccaa896955f"} Oct 30 20:00:01.991381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.991363 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerStarted","Data":"5d4a434953cdfe5daae569e66174863792c30e84c0f40debffbde6c78399b847"} Oct 30 20:00:01.991467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:01.991394 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerStarted","Data":"0c8eed1f3c00236d2be4d801c08d6e551821a1b9dca5526a7ad3ac23c7e0aed2"} Oct 30 20:00:02.017184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.017143 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" podStartSLOduration=2.017131252 podStartE2EDuration="2.017131252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.014799117 +0000 UTC m=+3309.726324483" watchObservedRunningTime="2026-10-30 20:00:02.017131252 +0000 UTC m=+3309.728656618" Oct 30 20:00:02.048668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.048617 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" podStartSLOduration=2.04860815 podStartE2EDuration="2.04860815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.04733245 +0000 UTC m=+3309.758857816" watchObservedRunningTime="2026-10-30 20:00:02.04860815 +0000 UTC m=+3309.760133516" Oct 30 20:00:02.087506 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.087461 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" podStartSLOduration=2.087447615 podStartE2EDuration="2.087447615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.083603847 +0000 UTC m=+3309.795129212" watchObservedRunningTime="2026-10-30 20:00:02.087447615 +0000 UTC m=+3309.798972980" Oct 30 20:00:02.123783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.123735 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" podStartSLOduration=2.123722483 podStartE2EDuration="2.123722483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.117454037 +0000 UTC m=+3309.828979404" watchObservedRunningTime="2026-10-30 20:00:02.123722483 +0000 UTC m=+3309.835247849" Oct 30 20:00:02.431732 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.431641 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5162479c-c704-4965-959c-fcc13f9e2259" path="/var/lib/kubelet/pods/5162479c-c704-4965-959c-fcc13f9e2259/volumes" Oct 30 20:00:02.433295 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:02.433264 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.005521 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.005490 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.007144 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.007118 4990 generic.go:356] "Generic (PLEG): container finished" podID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerID="516371b046cdcad219dae4d6ed64868f15b2e9e707a5fbd752b161a5d7a87c0a" exitCode=60 Oct 30 20:00:03.007278 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.007257 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" event={"ID":"c0c70006-7ab0-4e77-bd20-3689b18ebdb5","Type":"ContainerDied","Data":"516371b046cdcad219dae4d6ed64868f15b2e9e707a5fbd752b161a5d7a87c0a"} Oct 30 20:00:03.007886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.007853 4990 scope.go:122] "RemoveContainer" containerID="516371b046cdcad219dae4d6ed64868f15b2e9e707a5fbd752b161a5d7a87c0a" Oct 30 20:00:03.010842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.010813 4990 generic.go:356] "Generic (PLEG): container finished" podID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerID="9cd270a8f77b50f9ac7b2333d0aa0968dd8b37b7ab238439deac94bb10015ea0" exitCode=60 Oct 30 20:00:03.011023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.010956 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerDied","Data":"9cd270a8f77b50f9ac7b2333d0aa0968dd8b37b7ab238439deac94bb10015ea0"} Oct 30 20:00:03.011334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.011312 4990 scope.go:122] "RemoveContainer" containerID="9cd270a8f77b50f9ac7b2333d0aa0968dd8b37b7ab238439deac94bb10015ea0" Oct 30 20:00:03.014383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.014358 4990 generic.go:356] "Generic (PLEG): container finished" podID="914c666a-e17b-480d-8512-a90e7e8c165c" containerID="21abc37f3b800aa5b1cfab61af20770b953012fdbfb1a9f8127a8b8621425236" exitCode=60 Oct 30 20:00:03.014570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.014544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" event={"ID":"914c666a-e17b-480d-8512-a90e7e8c165c","Type":"ContainerDied","Data":"21abc37f3b800aa5b1cfab61af20770b953012fdbfb1a9f8127a8b8621425236"} Oct 30 20:00:03.014885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.014865 4990 scope.go:122] "RemoveContainer" containerID="21abc37f3b800aa5b1cfab61af20770b953012fdbfb1a9f8127a8b8621425236" Oct 30 20:00:03.018531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.018507 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerID="d487912484982bd8747ef4aadce4da38698be93f3dd3587c0c8c34e43e9cec40" exitCode=60 Oct 30 20:00:03.018620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.018537 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerDied","Data":"d487912484982bd8747ef4aadce4da38698be93f3dd3587c0c8c34e43e9cec40"} Oct 30 20:00:03.018917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.018898 4990 scope.go:122] "RemoveContainer" containerID="d487912484982bd8747ef4aadce4da38698be93f3dd3587c0c8c34e43e9cec40" Oct 30 20:00:03.022276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.021384 4990 generic.go:356] "Generic (PLEG): container finished" podID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerID="5a61ef1c68290bb17c1a33707ad93df83406d9fd6ff5b573897db36b82479661" exitCode=60 Oct 30 20:00:03.022276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.021446 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerDied","Data":"5a61ef1c68290bb17c1a33707ad93df83406d9fd6ff5b573897db36b82479661"} Oct 30 20:00:03.022276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.022032 4990 scope.go:122] "RemoveContainer" containerID="5a61ef1c68290bb17c1a33707ad93df83406d9fd6ff5b573897db36b82479661" Oct 30 20:00:03.024128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.024077 4990 generic.go:356] "Generic (PLEG): container finished" podID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerID="6e93c4960eafdb18ee9622835b13a3e4c65e5bb646afa9f6ef234981c052acf8" exitCode=60 Oct 30 20:00:03.024220 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.024150 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerDied","Data":"6e93c4960eafdb18ee9622835b13a3e4c65e5bb646afa9f6ef234981c052acf8"} Oct 30 20:00:03.024590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.024569 4990 scope.go:122] "RemoveContainer" containerID="6e93c4960eafdb18ee9622835b13a3e4c65e5bb646afa9f6ef234981c052acf8" Oct 30 20:00:03.026057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.026032 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e3425fb-eeaf-474a-9200-7018ac566400" containerID="da240b11c1b198b59bf1fba3061f46f697ff5b74f7f661b0c723107ce7605e8c" exitCode=60 Oct 30 20:00:03.026185 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.026150 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerDied","Data":"da240b11c1b198b59bf1fba3061f46f697ff5b74f7f661b0c723107ce7605e8c"} Oct 30 20:00:03.026463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.026444 4990 scope.go:122] "RemoveContainer" containerID="da240b11c1b198b59bf1fba3061f46f697ff5b74f7f661b0c723107ce7605e8c" Oct 30 20:00:03.036404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.036380 4990 generic.go:356] "Generic (PLEG): container finished" podID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerID="5d4a434953cdfe5daae569e66174863792c30e84c0f40debffbde6c78399b847" exitCode=60 Oct 30 20:00:03.036514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.036489 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerDied","Data":"5d4a434953cdfe5daae569e66174863792c30e84c0f40debffbde6c78399b847"} Oct 30 20:00:03.036705 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.036687 4990 scope.go:122] "RemoveContainer" containerID="5d4a434953cdfe5daae569e66174863792c30e84c0f40debffbde6c78399b847" Oct 30 20:00:03.039610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.038732 4990 generic.go:356] "Generic (PLEG): container finished" podID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerID="def6f2f4ba85f8e15b85a2daba9385df4ad2ee43e189e29145e23dbc85fe846b" exitCode=60 Oct 30 20:00:03.039610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.038836 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerDied","Data":"def6f2f4ba85f8e15b85a2daba9385df4ad2ee43e189e29145e23dbc85fe846b"} Oct 30 20:00:03.039610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.039116 4990 scope.go:122] "RemoveContainer" containerID="def6f2f4ba85f8e15b85a2daba9385df4ad2ee43e189e29145e23dbc85fe846b" Oct 30 20:00:03.041346 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.041326 4990 generic.go:356] "Generic (PLEG): container finished" podID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerID="be92be7bf6a4a492771790849b44b59bec870717571db9ce38b4a5b2072c5274" exitCode=60 Oct 30 20:00:03.041427 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.041404 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerDied","Data":"be92be7bf6a4a492771790849b44b59bec870717571db9ce38b4a5b2072c5274"} Oct 30 20:00:03.041972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.041952 4990 scope.go:122] "RemoveContainer" containerID="be92be7bf6a4a492771790849b44b59bec870717571db9ce38b4a5b2072c5274" Oct 30 20:00:03.049750 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.049709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerStarted","Data":"f75036a13ad07d3bb389b4c8d6fe2f917d06f624b8d06edd6eeaafb86627bea7"} Oct 30 20:00:03.295268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.295232 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.481771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.481725 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" podStartSLOduration=3.481709244 podStartE2EDuration="3.481709244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.477774263 +0000 UTC m=+3311.189299629" watchObservedRunningTime="2026-10-30 20:00:03.481709244 +0000 UTC m=+3311.193234676" Oct 30 20:00:03.817325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.814876 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 20:00:03.817325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:03.816746 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 20:00:04.078703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.076722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.078703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.077082 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" event={"ID":"c0c70006-7ab0-4e77-bd20-3689b18ebdb5","Type":"ContainerStarted","Data":"cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c"} Oct 30 20:00:04.096726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.094221 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerStarted","Data":"6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186"} Oct 30 20:00:04.117884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.116961 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerStarted","Data":"14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13"} Oct 30 20:00:04.126698 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.125894 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerStarted","Data":"663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf"} Oct 30 20:00:04.134862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.127780 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" podStartSLOduration=4.127761532 podStartE2EDuration="4.127761532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.119408615 +0000 UTC m=+3311.830933983" watchObservedRunningTime="2026-10-30 20:00:04.127761532 +0000 UTC m=+3311.839286903" Oct 30 20:00:04.137110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.137058 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerStarted","Data":"64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce"} Oct 30 20:00:04.139424 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.139404 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerStarted","Data":"412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd"} Oct 30 20:00:04.141486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.141462 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerStarted","Data":"bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1"} Oct 30 20:00:04.143596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.143576 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerStarted","Data":"2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec"} Oct 30 20:00:04.147645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.147624 4990 generic.go:356] "Generic (PLEG): container finished" podID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerID="f75036a13ad07d3bb389b4c8d6fe2f917d06f624b8d06edd6eeaafb86627bea7" exitCode=60 Oct 30 20:00:04.147744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.147709 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerDied","Data":"f75036a13ad07d3bb389b4c8d6fe2f917d06f624b8d06edd6eeaafb86627bea7"} Oct 30 20:00:04.147868 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.147854 4990 scope.go:122] "RemoveContainer" containerID="f75036a13ad07d3bb389b4c8d6fe2f917d06f624b8d06edd6eeaafb86627bea7" Oct 30 20:00:04.159649 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.159599 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" podStartSLOduration=4.159581827 podStartE2EDuration="4.159581827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.155973593 +0000 UTC m=+3311.867498958" watchObservedRunningTime="2026-10-30 20:00:04.159581827 +0000 UTC m=+3311.871107194" Oct 30 20:00:04.226034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.225981 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" podStartSLOduration=4.225963483 podStartE2EDuration="4.225963483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.223914474 +0000 UTC m=+3311.935439842" watchObservedRunningTime="2026-10-30 20:00:04.225963483 +0000 UTC m=+3311.937488845" Oct 30 20:00:04.227179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.227142 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:04.339990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.339937 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" podStartSLOduration=4.339921657 podStartE2EDuration="4.339921657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.334432199 +0000 UTC m=+3312.045957565" watchObservedRunningTime="2026-10-30 20:00:04.339921657 +0000 UTC m=+3312.051447023" Oct 30 20:00:04.376779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.376602 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" podStartSLOduration=4.376582155 podStartE2EDuration="4.376582155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.368659326 +0000 UTC m=+3312.080184693" watchObservedRunningTime="2026-10-30 20:00:04.376582155 +0000 UTC m=+3312.088107522" Oct 30 20:00:04.447744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:04.447695 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" podStartSLOduration=4.447680709 podStartE2EDuration="4.447680709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.444371135 +0000 UTC m=+3312.155896501" watchObservedRunningTime="2026-10-30 20:00:04.447680709 +0000 UTC m=+3312.159206074" Oct 30 20:00:05.098482 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.098448 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:00:05.160310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.160244 4990 generic.go:356] "Generic (PLEG): container finished" podID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerID="2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec" exitCode=60 Oct 30 20:00:05.160458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.160322 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerDied","Data":"2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec"} Oct 30 20:00:05.160458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.160392 4990 scope.go:122] "RemoveContainer" containerID="def6f2f4ba85f8e15b85a2daba9385df4ad2ee43e189e29145e23dbc85fe846b" Oct 30 20:00:05.160655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.160635 4990 scope.go:122] "RemoveContainer" containerID="2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec" Oct 30 20:00:05.160936 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.160917 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69_odh-ai-gateway-infra(04cb01c5-4e7f-4cd5-8bd4-08c3148341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" Oct 30 20:00:05.162784 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.162763 4990 generic.go:356] "Generic (PLEG): container finished" podID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerID="e218e7be2d50592e2c230d751eacea50b5df194078d9827091401ac71cfc7075" exitCode=60 Oct 30 20:00:05.162890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.162810 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerDied","Data":"e218e7be2d50592e2c230d751eacea50b5df194078d9827091401ac71cfc7075"} Oct 30 20:00:05.163188 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.163167 4990 scope.go:122] "RemoveContainer" containerID="e218e7be2d50592e2c230d751eacea50b5df194078d9827091401ac71cfc7075" Oct 30 20:00:05.163478 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.163455 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm_odh-ai-gateway-infra(782200a6-b9fd-491c-88e5-c0af7ab845d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" Oct 30 20:00:05.164874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.164847 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerStarted","Data":"2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37"} Oct 30 20:00:05.167251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.167226 4990 generic.go:356] "Generic (PLEG): container finished" podID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerID="cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c" exitCode=60 Oct 30 20:00:05.167359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.167303 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" event={"ID":"c0c70006-7ab0-4e77-bd20-3689b18ebdb5","Type":"ContainerDied","Data":"cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c"} Oct 30 20:00:05.167549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.167530 4990 scope.go:122] "RemoveContainer" containerID="cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c" Oct 30 20:00:05.167859 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.167837 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh_odh-ai-gateway-infra(c0c70006-7ab0-4e77-bd20-3689b18ebdb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" Oct 30 20:00:05.169418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.169396 4990 generic.go:356] "Generic (PLEG): container finished" podID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerID="6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186" exitCode=60 Oct 30 20:00:05.169678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.169664 4990 scope.go:122] "RemoveContainer" containerID="6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186" Oct 30 20:00:05.169917 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.169893 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52_odh-ai-gateway-infra(b8d9402f-122c-4206-a6e9-0ceda82c6117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" Oct 30 20:00:05.170158 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.170140 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerDied","Data":"6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186"} Oct 30 20:00:05.171514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.171495 4990 generic.go:356] "Generic (PLEG): container finished" podID="914c666a-e17b-480d-8512-a90e7e8c165c" containerID="c8f352661edb5c7498f47738399b7a34e83813210724f9324c2f98a2dcf00978" exitCode=60 Oct 30 20:00:05.171596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.171563 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" event={"ID":"914c666a-e17b-480d-8512-a90e7e8c165c","Type":"ContainerDied","Data":"c8f352661edb5c7498f47738399b7a34e83813210724f9324c2f98a2dcf00978"} Oct 30 20:00:05.171866 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.171846 4990 scope.go:122] "RemoveContainer" containerID="c8f352661edb5c7498f47738399b7a34e83813210724f9324c2f98a2dcf00978" Oct 30 20:00:05.172121 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.172085 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt_odh-ai-gateway-infra(914c666a-e17b-480d-8512-a90e7e8c165c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" Oct 30 20:00:05.179427 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.179399 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerID="14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13" exitCode=60 Oct 30 20:00:05.179541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.179476 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerDied","Data":"14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13"} Oct 30 20:00:05.179693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.179672 4990 scope.go:122] "RemoveContainer" containerID="14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13" Oct 30 20:00:05.179964 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.179935 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg_odh-ai-gateway-infra(1f599db9-6884-499a-b261-99ee7f7cd3a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" Oct 30 20:00:05.181167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.181141 4990 scope.go:122] "RemoveContainer" containerID="be92be7bf6a4a492771790849b44b59bec870717571db9ce38b4a5b2072c5274" Oct 30 20:00:05.181508 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.181489 4990 generic.go:356] "Generic (PLEG): container finished" podID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerID="663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf" exitCode=60 Oct 30 20:00:05.181595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.181572 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerDied","Data":"663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf"} Oct 30 20:00:05.182060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.182034 4990 scope.go:122] "RemoveContainer" containerID="663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf" Oct 30 20:00:05.182448 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.182287 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt_odh-ai-gateway-infra(62372dcf-ea83-4c7a-a3f9-96007c3aafcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" Oct 30 20:00:05.183614 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.183593 4990 generic.go:356] "Generic (PLEG): container finished" podID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerID="64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce" exitCode=60 Oct 30 20:00:05.183696 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.183674 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerDied","Data":"64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce"} Oct 30 20:00:05.183871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.183851 4990 scope.go:122] "RemoveContainer" containerID="64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce" Oct 30 20:00:05.184157 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.184081 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw_odh-ai-gateway-infra(ddf7c63a-b9c5-4532-8435-d61bcd804349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" Oct 30 20:00:05.185726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.185707 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e3425fb-eeaf-474a-9200-7018ac566400" containerID="412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd" exitCode=60 Oct 30 20:00:05.185800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.185764 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerDied","Data":"412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd"} Oct 30 20:00:05.185993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.185976 4990 scope.go:122] "RemoveContainer" containerID="412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd" Oct 30 20:00:05.186255 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.186235 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb_odh-ai-gateway-infra(0e3425fb-eeaf-474a-9200-7018ac566400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" Oct 30 20:00:05.187676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.187652 4990 generic.go:356] "Generic (PLEG): container finished" podID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerID="bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1" exitCode=60 Oct 30 20:00:05.187783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.187708 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerDied","Data":"bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1"} Oct 30 20:00:05.187930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.187914 4990 scope.go:122] "RemoveContainer" containerID="bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1" Oct 30 20:00:05.188153 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:05.188134 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh_odh-ai-gateway-infra(467b4713-0df1-4d0e-82bb-222aa22f7936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" Oct 30 20:00:05.196002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.195984 4990 scope.go:122] "RemoveContainer" containerID="516371b046cdcad219dae4d6ed64868f15b2e9e707a5fbd752b161a5d7a87c0a" Oct 30 20:00:05.210829 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.210808 4990 scope.go:122] "RemoveContainer" containerID="9cd270a8f77b50f9ac7b2333d0aa0968dd8b37b7ab238439deac94bb10015ea0" Oct 30 20:00:05.225073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.225053 4990 scope.go:122] "RemoveContainer" containerID="21abc37f3b800aa5b1cfab61af20770b953012fdbfb1a9f8127a8b8621425236" Oct 30 20:00:05.235363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.235340 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:05.239057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.239037 4990 scope.go:122] "RemoveContainer" containerID="d487912484982bd8747ef4aadce4da38698be93f3dd3587c0c8c34e43e9cec40" Oct 30 20:00:05.252834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.252808 4990 scope.go:122] "RemoveContainer" containerID="5a61ef1c68290bb17c1a33707ad93df83406d9fd6ff5b573897db36b82479661" Oct 30 20:00:05.268518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.268496 4990 scope.go:122] "RemoveContainer" containerID="6e93c4960eafdb18ee9622835b13a3e4c65e5bb646afa9f6ef234981c052acf8" Oct 30 20:00:05.284675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.284657 4990 scope.go:122] "RemoveContainer" containerID="da240b11c1b198b59bf1fba3061f46f697ff5b74f7f661b0c723107ce7605e8c" Oct 30 20:00:05.307386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:05.307359 4990 scope.go:122] "RemoveContainer" containerID="5d4a434953cdfe5daae569e66174863792c30e84c0f40debffbde6c78399b847" Oct 30 20:00:06.068114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.067625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:06.200824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.200786 4990 scope.go:122] "RemoveContainer" containerID="2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec" Oct 30 20:00:06.201344 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.201187 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69_odh-ai-gateway-infra(04cb01c5-4e7f-4cd5-8bd4-08c3148341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" Oct 30 20:00:06.204554 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.204525 4990 generic.go:356] "Generic (PLEG): container finished" podID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerID="2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37" exitCode=60 Oct 30 20:00:06.204687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.204611 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerDied","Data":"2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37"} Oct 30 20:00:06.204687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.204680 4990 scope.go:122] "RemoveContainer" containerID="f75036a13ad07d3bb389b4c8d6fe2f917d06f624b8d06edd6eeaafb86627bea7" Oct 30 20:00:06.204878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.204854 4990 scope.go:122] "RemoveContainer" containerID="2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37" Oct 30 20:00:06.205175 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.205153 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-mx55h_odh-ai-gateway-infra(596026b3-cdd7-4b4e-8416-d02c7b78364e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" Oct 30 20:00:06.207134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.207079 4990 scope.go:122] "RemoveContainer" containerID="cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c" Oct 30 20:00:06.207430 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.207402 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh_odh-ai-gateway-infra(c0c70006-7ab0-4e77-bd20-3689b18ebdb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" Oct 30 20:00:06.208835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.208810 4990 scope.go:122] "RemoveContainer" containerID="6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186" Oct 30 20:00:06.209128 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.209066 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52_odh-ai-gateway-infra(b8d9402f-122c-4206-a6e9-0ceda82c6117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" Oct 30 20:00:06.222874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.222848 4990 scope.go:122] "RemoveContainer" containerID="14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13" Oct 30 20:00:06.223338 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.223307 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg_odh-ai-gateway-infra(1f599db9-6884-499a-b261-99ee7f7cd3a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" Oct 30 20:00:06.225287 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.225266 4990 scope.go:122] "RemoveContainer" containerID="663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf" Oct 30 20:00:06.225720 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.225689 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt_odh-ai-gateway-infra(62372dcf-ea83-4c7a-a3f9-96007c3aafcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" Oct 30 20:00:06.227637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.227512 4990 scope.go:122] "RemoveContainer" containerID="64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce" Oct 30 20:00:06.227864 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.227843 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw_odh-ai-gateway-infra(ddf7c63a-b9c5-4532-8435-d61bcd804349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" Oct 30 20:00:06.229139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.229120 4990 scope.go:122] "RemoveContainer" containerID="412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd" Oct 30 20:00:06.229393 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.229370 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb_odh-ai-gateway-infra(0e3425fb-eeaf-474a-9200-7018ac566400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" Oct 30 20:00:06.230693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:06.230673 4990 scope.go:122] "RemoveContainer" containerID="bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1" Oct 30 20:00:06.230910 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:06.230889 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh_odh-ai-gateway-infra(467b4713-0df1-4d0e-82bb-222aa22f7936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" Oct 30 20:00:07.241165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:07.241136 4990 scope.go:122] "RemoveContainer" containerID="2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37" Oct 30 20:00:07.241580 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:07.241393 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-mx55h_odh-ai-gateway-infra(596026b3-cdd7-4b4e-8416-d02c7b78364e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" Oct 30 20:00:08.130374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:08.130336 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:08.278792 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:08.278752 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:09.596801 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.596767 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:09.642799 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.642759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:09.828116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.823852 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:09.974168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.973470 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\") pod \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " Oct 30 20:00:09.974168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.973662 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\" (UniqueName: \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\") pod \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\" (UID: \"c0c70006-7ab0-4e77-bd20-3689b18ebdb5\") " Oct 30 20:00:09.975130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.974868 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca" pod "c0c70006-7ab0-4e77-bd20-3689b18ebdb5" (UID: "c0c70006-7ab0-4e77-bd20-3689b18ebdb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:09.992351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:09.981304 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc" pod "c0c70006-7ab0-4e77-bd20-3689b18ebdb5" (UID: "c0c70006-7ab0-4e77-bd20-3689b18ebdb5"). InnerVolumeSpecName "kube-api-access-m2cbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:10.045176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.045144 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9"] Oct 30 20:00:10.053038 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.052998 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e5c5a3-4bjq9"] Oct 30 20:00:10.097113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.096584 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.097113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.096636 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cbc\" (UniqueName: \"kubernetes.io/projected/c0c70006-7ab0-4e77-bd20-3689b18ebdb5-kube-api-access-m2cbc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.159890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.159850 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.271176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.271054 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" Oct 30 20:00:10.271176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.271084 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh" event={"ID":"c0c70006-7ab0-4e77-bd20-3689b18ebdb5","Type":"ContainerDied","Data":"d6dc97d44a576d7133372b7ccf422b62e09eca4386e4746dbb42da96f02fb554"} Oct 30 20:00:10.271403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.271194 4990 scope.go:122] "RemoveContainer" containerID="cbb542cfddab009cbeb526cb6b647199c1669e2ece2258f0197ebb503c44955c" Oct 30 20:00:10.288477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.288455 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9"] Oct 30 20:00:10.297296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.297271 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a56cb5-nf4t9"] Oct 30 20:00:10.323936 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.323910 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:10.329862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.329839 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a56cb5-29889840-tgrsh"] Oct 30 20:00:10.331822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.331798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.405907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.405860 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aacf09c-465b-4e69-bf1f-59c3fe19370b" path="/var/lib/kubelet/pods/5aacf09c-465b-4e69-bf1f-59c3fe19370b/volumes" Oct 30 20:00:10.406460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.406434 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" path="/var/lib/kubelet/pods/c0c70006-7ab0-4e77-bd20-3689b18ebdb5/volumes" Oct 30 20:00:10.406989 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.406966 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed008c61-7716-42f4-a131-a45aa525da2f" path="/var/lib/kubelet/pods/ed008c61-7716-42f4-a131-a45aa525da2f/volumes" Oct 30 20:00:10.478518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.478477 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.935080 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:10.935040 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.034857 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:11.034825 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk"] Oct 30 20:00:11.039741 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:11.039720 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc22d0-58jfk"] Oct 30 20:00:12.061636 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.061612 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 20:00:12.065203 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.065183 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 20:00:12.167418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.167374 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.229775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.229736 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\") pod \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " Oct 30 20:00:12.229977 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.229833 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\") pod \"5468dfab-1958-4793-a2e3-f83df3930d12\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " Oct 30 20:00:12.230067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.229992 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\" (UniqueName: \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\") pod \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\" (UID: \"1564f703-b9e7-4851-8ddd-e1f6b7b0a401\") " Oct 30 20:00:12.230209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.230187 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\" (UniqueName: \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\") pod \"5468dfab-1958-4793-a2e3-f83df3930d12\" (UID: \"5468dfab-1958-4793-a2e3-f83df3930d12\") " Oct 30 20:00:12.230332 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.230303 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume" pod "1564f703-b9e7-4851-8ddd-e1f6b7b0a401" (UID: "1564f703-b9e7-4851-8ddd-e1f6b7b0a401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:12.230332 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.230315 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume" pod "5468dfab-1958-4793-a2e3-f83df3930d12" (UID: "5468dfab-1958-4793-a2e3-f83df3930d12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:12.230842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.230818 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5468dfab-1958-4793-a2e3-f83df3930d12-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:12.230927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.230845 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:12.232361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.232331 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85" pod "1564f703-b9e7-4851-8ddd-e1f6b7b0a401" (UID: "1564f703-b9e7-4851-8ddd-e1f6b7b0a401"). InnerVolumeSpecName "kube-api-access-kjn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:12.232453 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.232408 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w" pod "5468dfab-1958-4793-a2e3-f83df3930d12" (UID: "5468dfab-1958-4793-a2e3-f83df3930d12"). InnerVolumeSpecName "kube-api-access-m4r7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:12.297196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.297080 4990 generic.go:356] "Generic (PLEG): container finished" podID="5468dfab-1958-4793-a2e3-f83df3930d12" containerID="1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791" exitCode=0 Oct 30 20:00:12.297196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.297123 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" event={"ID":"5468dfab-1958-4793-a2e3-f83df3930d12","Type":"ContainerDied","Data":"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791"} Oct 30 20:00:12.297196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.297166 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" Oct 30 20:00:12.297196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.297188 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47" event={"ID":"5468dfab-1958-4793-a2e3-f83df3930d12","Type":"ContainerDied","Data":"57a1ecb5fbafc46eccff33b6e8a6a0dd1678245b3285efbe86205f29b976916f"} Oct 30 20:00:12.297568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.297218 4990 scope.go:122] "RemoveContainer" containerID="1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791" Oct 30 20:00:12.298795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.298773 4990 generic.go:356] "Generic (PLEG): container finished" podID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerID="cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856" exitCode=0 Oct 30 20:00:12.299051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.299033 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" event={"ID":"1564f703-b9e7-4851-8ddd-e1f6b7b0a401","Type":"ContainerDied","Data":"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856"} Oct 30 20:00:12.299183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.299080 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" event={"ID":"1564f703-b9e7-4851-8ddd-e1f6b7b0a401","Type":"ContainerDied","Data":"f3887770f68cbfb5538bb56e302017826e5884c55cf52d735a11bf54716addd2"} Oct 30 20:00:12.299266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.299194 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x" Oct 30 20:00:12.313230 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.313211 4990 scope.go:122] "RemoveContainer" containerID="1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791" Oct 30 20:00:12.313556 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:12.313534 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791\": container with ID starting with 1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791 not found: ID does not exist" containerID="1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791" Oct 30 20:00:12.313643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.313567 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791"} err="failed to get container status \"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791\": rpc error: code = NotFound desc = could not find container \"1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791\": container with ID starting with 1bfba4c4f19143c886fed134595ca6e54ef52ae8c92eb00df23d5eb122da4791 not found: ID does not exist" Oct 30 20:00:12.313643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.313590 4990 scope.go:122] "RemoveContainer" containerID="cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856" Oct 30 20:00:12.324167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.324152 4990 scope.go:122] "RemoveContainer" containerID="cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856" Oct 30 20:00:12.324424 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:12.324395 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856\": container with ID starting with cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856 not found: ID does not exist" containerID="cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856" Oct 30 20:00:12.324507 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.324426 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856"} err="failed to get container status \"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856\": rpc error: code = NotFound desc = could not find container \"cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856\": container with ID starting with cf1c33925560d4ecb7afc06e31a0f94cd4adf042e9bbc219c6064eb95dedb856 not found: ID does not exist" Oct 30 20:00:12.332015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.331991 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjn85\" (UniqueName: \"kubernetes.io/projected/1564f703-b9e7-4851-8ddd-e1f6b7b0a401-kube-api-access-kjn85\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:12.332116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.332016 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4r7w\" (UniqueName: \"kubernetes.io/projected/5468dfab-1958-4793-a2e3-f83df3930d12-kube-api-access-m4r7w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:12.356995 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.356972 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 20:00:12.361514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.361494 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a56cb5-6dbfd8c499-6gw8x"] Oct 30 20:00:12.388036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.388003 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 20:00:12.405546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.405515 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" path="/var/lib/kubelet/pods/1564f703-b9e7-4851-8ddd-e1f6b7b0a401/volumes" Oct 30 20:00:12.406005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.405989 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0ef85f-7a36-4caa-8a9a-6c47313a4e38" path="/var/lib/kubelet/pods/3f0ef85f-7a36-4caa-8a9a-6c47313a4e38/volumes" Oct 30 20:00:12.406383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:12.406366 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a56cb5-85f6878cc8-s8p47"] Oct 30 20:00:13.931667 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:13.930130 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.406268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:14.406226 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" path="/var/lib/kubelet/pods/5468dfab-1958-4793-a2e3-f83df3930d12/volumes" Oct 30 20:00:15.393839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:15.393811 4990 scope.go:122] "RemoveContainer" containerID="663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf" Oct 30 20:00:15.394283 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:15.393955 4990 scope.go:122] "RemoveContainer" containerID="6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186" Oct 30 20:00:15.556951 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:15.556926 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.348356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.348325 4990 generic.go:356] "Generic (PLEG): container finished" podID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerID="717f8a00b93862672058a8fa2a3f3b0268a079470d92f7df8da547d814de4b84" exitCode=60 Oct 30 20:00:16.348493 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.348398 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerDied","Data":"717f8a00b93862672058a8fa2a3f3b0268a079470d92f7df8da547d814de4b84"} Oct 30 20:00:16.348493 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.348462 4990 scope.go:122] "RemoveContainer" containerID="6dd2d17df31b0322618c42ebd3599eaa89622956cccd75eb390ac62883f7c186" Oct 30 20:00:16.348879 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.348861 4990 scope.go:122] "RemoveContainer" containerID="717f8a00b93862672058a8fa2a3f3b0268a079470d92f7df8da547d814de4b84" Oct 30 20:00:16.349294 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:16.349265 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52_odh-ai-gateway-infra(b8d9402f-122c-4206-a6e9-0ceda82c6117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" Oct 30 20:00:16.351056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.351029 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerStarted","Data":"29d8d9ea55c545e2771a198f5f1c37407772a5e775478c7a0aa64f789dd0b1fe"} Oct 30 20:00:16.393726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:16.393700 4990 scope.go:122] "RemoveContainer" containerID="64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce" Oct 30 20:00:17.274734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.274177 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.363853 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.363817 4990 generic.go:356] "Generic (PLEG): container finished" podID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerID="29d8d9ea55c545e2771a198f5f1c37407772a5e775478c7a0aa64f789dd0b1fe" exitCode=60 Oct 30 20:00:17.364023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.363890 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerDied","Data":"29d8d9ea55c545e2771a198f5f1c37407772a5e775478c7a0aa64f789dd0b1fe"} Oct 30 20:00:17.364023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.363997 4990 scope.go:122] "RemoveContainer" containerID="663c2ba8c7c89b1faf4db5d87e896903325c9ff22941b055a61acb9575086caf" Oct 30 20:00:17.364423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.364401 4990 scope.go:122] "RemoveContainer" containerID="29d8d9ea55c545e2771a198f5f1c37407772a5e775478c7a0aa64f789dd0b1fe" Oct 30 20:00:17.364854 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:17.364788 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt_odh-ai-gateway-infra(62372dcf-ea83-4c7a-a3f9-96007c3aafcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" Oct 30 20:00:17.370247 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.370219 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerStarted","Data":"e5d37b02652d6d533e583f620d2cd9a3826377b0ae8f3300be494ecf887329bd"} Oct 30 20:00:17.395243 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.395206 4990 scope.go:122] "RemoveContainer" containerID="bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1" Oct 30 20:00:17.396690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.396658 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:17.436856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.436687 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:17.652149 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.646610 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:17.756141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.756108 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\") pod \"b8d9402f-122c-4206-a6e9-0ceda82c6117\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " Oct 30 20:00:17.756316 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.756170 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\" (UniqueName: \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\") pod \"b8d9402f-122c-4206-a6e9-0ceda82c6117\" (UID: \"b8d9402f-122c-4206-a6e9-0ceda82c6117\") " Oct 30 20:00:17.756487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.756461 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca" pod "b8d9402f-122c-4206-a6e9-0ceda82c6117" (UID: "b8d9402f-122c-4206-a6e9-0ceda82c6117"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.757239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.757213 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d9402f-122c-4206-a6e9-0ceda82c6117-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:17.758471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.758441 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw" pod "b8d9402f-122c-4206-a6e9-0ceda82c6117" (UID: "b8d9402f-122c-4206-a6e9-0ceda82c6117"). InnerVolumeSpecName "kube-api-access-4hdrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.859444 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:17.859367 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdrw\" (UniqueName: \"kubernetes.io/projected/b8d9402f-122c-4206-a6e9-0ceda82c6117-kube-api-access-4hdrw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:18.164044 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.163958 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.319937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.319891 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.396136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.396030 4990 scope.go:122] "RemoveContainer" containerID="e218e7be2d50592e2c230d751eacea50b5df194078d9827091401ac71cfc7075" Oct 30 20:00:18.400219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.400197 4990 scope.go:122] "RemoveContainer" containerID="2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37" Oct 30 20:00:18.402421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.401956 4990 scope.go:122] "RemoveContainer" containerID="412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd" Oct 30 20:00:18.402421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.402324 4990 scope.go:122] "RemoveContainer" containerID="14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13" Oct 30 20:00:18.402640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.402502 4990 scope.go:122] "RemoveContainer" containerID="2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec" Oct 30 20:00:18.402783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.402767 4990 scope.go:122] "RemoveContainer" containerID="c8f352661edb5c7498f47738399b7a34e83813210724f9324c2f98a2dcf00978" Oct 30 20:00:18.462378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.460172 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 20:00:18.462378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.460388 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" containerID="cri-o://5e219ec191cdbafce96d7915da1fcb353f17ed81377827d6a978d5bfec68e37a" gracePeriod=30 Oct 30 20:00:18.485786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.483370 4990 generic.go:356] "Generic (PLEG): container finished" podID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerID="e5d37b02652d6d533e583f620d2cd9a3826377b0ae8f3300be494ecf887329bd" exitCode=60 Oct 30 20:00:18.485786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.483483 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerDied","Data":"e5d37b02652d6d533e583f620d2cd9a3826377b0ae8f3300be494ecf887329bd"} Oct 30 20:00:18.485786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.483533 4990 scope.go:122] "RemoveContainer" containerID="64bbc917ba1a285d3d30ceab9d02fb803bedbb543d04f704ff3c1f225dafdbce" Oct 30 20:00:18.485786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.484074 4990 scope.go:122] "RemoveContainer" containerID="e5d37b02652d6d533e583f620d2cd9a3826377b0ae8f3300be494ecf887329bd" Oct 30 20:00:18.485786 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:18.484376 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw_odh-ai-gateway-infra(ddf7c63a-b9c5-4532-8435-d61bcd804349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" Oct 30 20:00:18.493181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.493149 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerStarted","Data":"295f31d3e13257f349b3deb49be30a5f6bb185fb8ca264ff8f89be43fbd9681a"} Oct 30 20:00:18.497534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.496892 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" Oct 30 20:00:18.497534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.497428 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:18.497534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.497467 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52" event={"ID":"b8d9402f-122c-4206-a6e9-0ceda82c6117","Type":"ContainerDied","Data":"a0b82453ee8e924bbc50ed759f323ac5669708603c68baee10dac2e06fb6ba13"} Oct 30 20:00:18.521280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.521183 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.785435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.784399 4990 scope.go:122] "RemoveContainer" containerID="717f8a00b93862672058a8fa2a3f3b0268a079470d92f7df8da547d814de4b84" Oct 30 20:00:18.798455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.798425 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:18.806414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.806370 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889840-drw52"] Oct 30 20:00:18.873860 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.873787 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.875626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.875606 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:18.981410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.981378 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 20:00:18.981647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:18.981616 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" podUID="ff27e004-cb42-4336-81cb-9881e663796b" containerName="payload-pre-processing" containerID="cri-o://052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea" gracePeriod=30 Oct 30 20:00:19.011126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.011064 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\" (UniqueName: \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\") pod \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " Oct 30 20:00:19.011249 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.011234 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\") pod \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\" (UID: \"62372dcf-ea83-4c7a-a3f9-96007c3aafcb\") " Oct 30 20:00:19.014061 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.014032 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca" pod "62372dcf-ea83-4c7a-a3f9-96007c3aafcb" (UID: "62372dcf-ea83-4c7a-a3f9-96007c3aafcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.087681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.087642 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.113880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.113857 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.169361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.169332 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q" pod "62372dcf-ea83-4c7a-a3f9-96007c3aafcb" (UID: "62372dcf-ea83-4c7a-a3f9-96007c3aafcb"). InnerVolumeSpecName "kube-api-access-jf85q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.215236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.215084 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf85q\" (UniqueName: \"kubernetes.io/projected/62372dcf-ea83-4c7a-a3f9-96007c3aafcb-kube-api-access-jf85q\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.240421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.240371 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.468275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.461294 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 20:00:19.537111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.536456 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" event={"ID":"62372dcf-ea83-4c7a-a3f9-96007c3aafcb","Type":"ContainerDied","Data":"8cd17bbaa57e5838f32339d9502615f649958f8224ecca13d54e6d18daa6a1a3"} Oct 30 20:00:19.537111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.536530 4990 scope.go:122] "RemoveContainer" containerID="29d8d9ea55c545e2771a198f5f1c37407772a5e775478c7a0aa64f789dd0b1fe" Oct 30 20:00:19.537111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.536636 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt" Oct 30 20:00:19.578822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.573686 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\" (UniqueName: \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\") pod \"ff27e004-cb42-4336-81cb-9881e663796b\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " Oct 30 20:00:19.579470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.579438 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\") pod \"ff27e004-cb42-4336-81cb-9881e663796b\" (UID: \"ff27e004-cb42-4336-81cb-9881e663796b\") " Oct 30 20:00:19.594241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.592541 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume" pod "ff27e004-cb42-4336-81cb-9881e663796b" (UID: "ff27e004-cb42-4336-81cb-9881e663796b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.597362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.597328 4990 generic.go:356] "Generic (PLEG): container finished" podID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerID="295f31d3e13257f349b3deb49be30a5f6bb185fb8ca264ff8f89be43fbd9681a" exitCode=60 Oct 30 20:00:19.597501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.597454 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerDied","Data":"295f31d3e13257f349b3deb49be30a5f6bb185fb8ca264ff8f89be43fbd9681a"} Oct 30 20:00:19.597905 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.597884 4990 scope.go:122] "RemoveContainer" containerID="295f31d3e13257f349b3deb49be30a5f6bb185fb8ca264ff8f89be43fbd9681a" Oct 30 20:00:19.603606 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.603561 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk" pod "ff27e004-cb42-4336-81cb-9881e663796b" (UID: "ff27e004-cb42-4336-81cb-9881e663796b"). InnerVolumeSpecName "kube-api-access-mn2bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.609905 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:19.609029 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh_odh-ai-gateway-infra(467b4713-0df1-4d0e-82bb-222aa22f7936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" Oct 30 20:00:19.617503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.617484 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerStarted","Data":"17abace3f27d336a95b03fff65123c547e77ae5e333cbd3a67498406f020119f"} Oct 30 20:00:19.623615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.623597 4990 generic.go:356] "Generic (PLEG): container finished" podID="ff27e004-cb42-4336-81cb-9881e663796b" containerID="052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea" exitCode=0 Oct 30 20:00:19.623782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.623767 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" event={"ID":"ff27e004-cb42-4336-81cb-9881e663796b","Type":"ContainerDied","Data":"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea"} Oct 30 20:00:19.623874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.623864 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" event={"ID":"ff27e004-cb42-4336-81cb-9881e663796b","Type":"ContainerDied","Data":"01a31f041ff40c4fcfbf91a3665cc1b65ad4c2d5ef482c7b7797b082ab286d5b"} Oct 30 20:00:19.623974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.623965 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9" Oct 30 20:00:19.653437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.652825 4990 scope.go:122] "RemoveContainer" containerID="bcc4ca4d44947710cbed2ecb4b7939ee330d30c243970929504fa372c463feb1" Oct 30 20:00:19.680636 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.680610 4990 scope.go:122] "RemoveContainer" containerID="052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea" Oct 30 20:00:19.682683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.682487 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff27e004-cb42-4336-81cb-9881e663796b-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.682683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.682529 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn2bk\" (UniqueName: \"kubernetes.io/projected/ff27e004-cb42-4336-81cb-9881e663796b-kube-api-access-mn2bk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.734805 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.734725 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:19.741110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.741065 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-5a0a17-29889840-cbgtt"] Oct 30 20:00:19.750244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.750214 4990 scope.go:122] "RemoveContainer" containerID="052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea" Oct 30 20:00:19.750822 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:19.750705 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea\": container with ID starting with 052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea not found: ID does not exist" containerID="052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea" Oct 30 20:00:19.750822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.750747 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea"} err="failed to get container status \"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea\": rpc error: code = NotFound desc = could not find container \"052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea\": container with ID starting with 052233d17481b482ee6c71267ffa1e5042e2d3b26f1e22099d4fa5df36722aea not found: ID does not exist" Oct 30 20:00:19.771471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.771281 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:19.772731 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.772595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.804529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.804493 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 20:00:19.812495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.812470 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf314f-78d6bdf8f9-tl2s9"] Oct 30 20:00:19.887360 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.887281 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb"] Oct 30 20:00:19.892040 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.892007 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf314f-d6mcb"] Oct 30 20:00:19.904011 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.903983 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:19.939054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:19.939013 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.023366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.023325 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\" (UniqueName: \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\") pod \"ddf7c63a-b9c5-4532-8435-d61bcd804349\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " Oct 30 20:00:20.023577 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.023478 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\") pod \"ddf7c63a-b9c5-4532-8435-d61bcd804349\" (UID: \"ddf7c63a-b9c5-4532-8435-d61bcd804349\") " Oct 30 20:00:20.027636 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.027603 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca" pod "ddf7c63a-b9c5-4532-8435-d61bcd804349" (UID: "ddf7c63a-b9c5-4532-8435-d61bcd804349"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.031420 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.031386 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm" pod "ddf7c63a-b9c5-4532-8435-d61bcd804349" (UID: "ddf7c63a-b9c5-4532-8435-d61bcd804349"). InnerVolumeSpecName "kube-api-access-dkzzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.033900 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.033873 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr"] Oct 30 20:00:20.038918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.038894 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caa49937-pxqkr"] Oct 30 20:00:20.089561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.089531 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.125972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.125941 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzzm\" (UniqueName: \"kubernetes.io/projected/ddf7c63a-b9c5-4532-8435-d61bcd804349-kube-api-access-dkzzm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.126076 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.125977 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf7c63a-b9c5-4532-8435-d61bcd804349-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.403472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.403439 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" path="/var/lib/kubelet/pods/62372dcf-ea83-4c7a-a3f9-96007c3aafcb/volumes" Oct 30 20:00:20.403931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.403916 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8c3f45-8ad0-46f0-9ef3-ef534660f91e" path="/var/lib/kubelet/pods/8a8c3f45-8ad0-46f0-9ef3-ef534660f91e/volumes" Oct 30 20:00:20.404343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.404329 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" path="/var/lib/kubelet/pods/b8d9402f-122c-4206-a6e9-0ceda82c6117/volumes" Oct 30 20:00:20.405132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.405085 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe45fab4-338c-41e6-9515-1168d477466c" path="/var/lib/kubelet/pods/fe45fab4-338c-41e6-9515-1168d477466c/volumes" Oct 30 20:00:20.405511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.405498 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff27e004-cb42-4336-81cb-9881e663796b" path="/var/lib/kubelet/pods/ff27e004-cb42-4336-81cb-9881e663796b/volumes" Oct 30 20:00:20.469518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.469490 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.636574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.636539 4990 generic.go:356] "Generic (PLEG): container finished" podID="914c666a-e17b-480d-8512-a90e7e8c165c" containerID="c3bbe21b83331467a5b76433b3720a9ea9a809fc2d2d8df5fe1307378b543d44" exitCode=60 Oct 30 20:00:20.636721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.636641 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" event={"ID":"914c666a-e17b-480d-8512-a90e7e8c165c","Type":"ContainerDied","Data":"c3bbe21b83331467a5b76433b3720a9ea9a809fc2d2d8df5fe1307378b543d44"} Oct 30 20:00:20.636721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.636685 4990 scope.go:122] "RemoveContainer" containerID="c8f352661edb5c7498f47738399b7a34e83813210724f9324c2f98a2dcf00978" Oct 30 20:00:20.637051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.637033 4990 scope.go:122] "RemoveContainer" containerID="c3bbe21b83331467a5b76433b3720a9ea9a809fc2d2d8df5fe1307378b543d44" Oct 30 20:00:20.637323 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.637297 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt_odh-ai-gateway-infra(914c666a-e17b-480d-8512-a90e7e8c165c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" Oct 30 20:00:20.639857 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.639835 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerID="e0527e483f435d0f8242100fc06ad2a3cf9d76fc092e030823c65111c37cbda4" exitCode=60 Oct 30 20:00:20.639939 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.639904 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerDied","Data":"e0527e483f435d0f8242100fc06ad2a3cf9d76fc092e030823c65111c37cbda4"} Oct 30 20:00:20.640248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.640232 4990 scope.go:122] "RemoveContainer" containerID="e0527e483f435d0f8242100fc06ad2a3cf9d76fc092e030823c65111c37cbda4" Oct 30 20:00:20.640482 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.640464 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg_odh-ai-gateway-infra(1f599db9-6884-499a-b261-99ee7f7cd3a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" Oct 30 20:00:20.642263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.642247 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" Oct 30 20:00:20.642336 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.642260 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw" event={"ID":"ddf7c63a-b9c5-4532-8435-d61bcd804349","Type":"ContainerDied","Data":"d838b3ab55ea9b4c4b27173f2e5305e1da7c48fa720ac382ad37158769777e9e"} Oct 30 20:00:20.644541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.644517 4990 generic.go:356] "Generic (PLEG): container finished" podID="0e3425fb-eeaf-474a-9200-7018ac566400" containerID="7131ce336c847f547a53db08dcd9e56c28717259e865cbfd456bf54a9866e641" exitCode=60 Oct 30 20:00:20.644640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.644592 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerDied","Data":"7131ce336c847f547a53db08dcd9e56c28717259e865cbfd456bf54a9866e641"} Oct 30 20:00:20.644928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.644908 4990 scope.go:122] "RemoveContainer" containerID="7131ce336c847f547a53db08dcd9e56c28717259e865cbfd456bf54a9866e641" Oct 30 20:00:20.645207 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.645184 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb_odh-ai-gateway-infra(0e3425fb-eeaf-474a-9200-7018ac566400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" Oct 30 20:00:20.648465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.648445 4990 generic.go:356] "Generic (PLEG): container finished" podID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerID="99cbc2d34ff474e62332dcc55f49231f4ff277cf266e8c9cedefa4a411681cd2" exitCode=60 Oct 30 20:00:20.648574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.648516 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerDied","Data":"99cbc2d34ff474e62332dcc55f49231f4ff277cf266e8c9cedefa4a411681cd2"} Oct 30 20:00:20.648866 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.648848 4990 scope.go:122] "RemoveContainer" containerID="99cbc2d34ff474e62332dcc55f49231f4ff277cf266e8c9cedefa4a411681cd2" Oct 30 20:00:20.649175 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.649149 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69_odh-ai-gateway-infra(04cb01c5-4e7f-4cd5-8bd4-08c3148341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" Oct 30 20:00:20.650595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.650571 4990 generic.go:356] "Generic (PLEG): container finished" podID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerID="17abace3f27d336a95b03fff65123c547e77ae5e333cbd3a67498406f020119f" exitCode=60 Oct 30 20:00:20.650688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.650597 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerDied","Data":"17abace3f27d336a95b03fff65123c547e77ae5e333cbd3a67498406f020119f"} Oct 30 20:00:20.650889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.650873 4990 scope.go:122] "RemoveContainer" containerID="17abace3f27d336a95b03fff65123c547e77ae5e333cbd3a67498406f020119f" Oct 30 20:00:20.651141 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.651118 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm_odh-ai-gateway-infra(782200a6-b9fd-491c-88e5-c0af7ab845d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" Oct 30 20:00:20.652516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.652497 4990 generic.go:356] "Generic (PLEG): container finished" podID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerID="af8f72e34a49d79f3fb1b70736f72a1863e778e26690d1ec7d83b93128ca4594" exitCode=60 Oct 30 20:00:20.652601 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.652565 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerDied","Data":"af8f72e34a49d79f3fb1b70736f72a1863e778e26690d1ec7d83b93128ca4594"} Oct 30 20:00:20.652923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.652905 4990 scope.go:122] "RemoveContainer" containerID="af8f72e34a49d79f3fb1b70736f72a1863e778e26690d1ec7d83b93128ca4594" Oct 30 20:00:20.653183 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:20.653160 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-mx55h_odh-ai-gateway-infra(596026b3-cdd7-4b4e-8416-d02c7b78364e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" Oct 30 20:00:20.656591 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.656573 4990 scope.go:122] "RemoveContainer" containerID="14eaf9da5e2b2dcf7ad63621df4d23b0c2eb61b8eef1801502edd3ad4ad2cb13" Oct 30 20:00:20.673953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.673933 4990 scope.go:122] "RemoveContainer" containerID="e5d37b02652d6d533e583f620d2cd9a3826377b0ae8f3300be494ecf887329bd" Oct 30 20:00:20.686168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.686069 4990 scope.go:122] "RemoveContainer" containerID="412b829fa0fb620eeb6c7763e165e85e7dc674f8920d8565a8792aba467c24dd" Oct 30 20:00:20.704308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.704289 4990 scope.go:122] "RemoveContainer" containerID="2eea0cf083a6db3bbcd4a9b9f44b4347f4bf20ae23208b0f3c4df8bee323cbec" Oct 30 20:00:20.719665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.718813 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:20.734156 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.733244 4990 scope.go:122] "RemoveContainer" containerID="e218e7be2d50592e2c230d751eacea50b5df194078d9827091401ac71cfc7075" Oct 30 20:00:20.758944 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.754289 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:20.776988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.774079 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-315ea2-29889840-8jfqw"] Oct 30 20:00:20.782602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.782278 4990 scope.go:122] "RemoveContainer" containerID="2a07955e188443dbd14cf4ae8c00eef9c192cc1ce9908c4d0f31d206136d4b37" Oct 30 20:00:20.860679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.860656 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:20.940425 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.940392 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\") pod \"467b4713-0df1-4d0e-82bb-222aa22f7936\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " Oct 30 20:00:20.940545 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.940512 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\" (UniqueName: \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\") pod \"467b4713-0df1-4d0e-82bb-222aa22f7936\" (UID: \"467b4713-0df1-4d0e-82bb-222aa22f7936\") " Oct 30 20:00:20.940708 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.940687 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca" pod "467b4713-0df1-4d0e-82bb-222aa22f7936" (UID: "467b4713-0df1-4d0e-82bb-222aa22f7936"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.941078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.941051 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467b4713-0df1-4d0e-82bb-222aa22f7936-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.942652 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:20.942622 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm" pod "467b4713-0df1-4d0e-82bb-222aa22f7936" (UID: "467b4713-0df1-4d0e-82bb-222aa22f7936"). InnerVolumeSpecName "kube-api-access-9tksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.051766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.042371 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tksm\" (UniqueName: \"kubernetes.io/projected/467b4713-0df1-4d0e-82bb-222aa22f7936-kube-api-access-9tksm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:21.424358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.424274 4990 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:00:21.424358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.424332 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:00:21.674631 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.674544 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" Oct 30 20:00:21.674631 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.674579 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh" event={"ID":"467b4713-0df1-4d0e-82bb-222aa22f7936","Type":"ContainerDied","Data":"0c8eed1f3c00236d2be4d801c08d6e551821a1b9dca5526a7ad3ac23c7e0aed2"} Oct 30 20:00:21.675149 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.674656 4990 scope.go:122] "RemoveContainer" containerID="295f31d3e13257f349b3deb49be30a5f6bb185fb8ca264ff8f89be43fbd9681a" Oct 30 20:00:21.694979 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.694949 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:21.786292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.786243 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:21.791307 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.791280 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ed2ede-29889840-8ldzh"] Oct 30 20:00:21.886905 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.886876 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:21.892250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.892222 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:21.908859 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.908830 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:21.937169 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.937103 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:21.954445 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.954418 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\" (UniqueName: \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\") pod \"914c666a-e17b-480d-8512-a90e7e8c165c\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " Oct 30 20:00:21.954627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.954469 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\" (UniqueName: \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\") pod \"782200a6-b9fd-491c-88e5-c0af7ab845d1\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " Oct 30 20:00:21.954627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.954559 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\") pod \"782200a6-b9fd-491c-88e5-c0af7ab845d1\" (UID: \"782200a6-b9fd-491c-88e5-c0af7ab845d1\") " Oct 30 20:00:21.954627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.954586 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\") pod \"914c666a-e17b-480d-8512-a90e7e8c165c\" (UID: \"914c666a-e17b-480d-8512-a90e7e8c165c\") " Oct 30 20:00:21.955007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.954984 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca" pod "782200a6-b9fd-491c-88e5-c0af7ab845d1" (UID: "782200a6-b9fd-491c-88e5-c0af7ab845d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.955407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.955381 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca" pod "914c666a-e17b-480d-8512-a90e7e8c165c" (UID: "914c666a-e17b-480d-8512-a90e7e8c165c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.957422 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.957392 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6" pod "782200a6-b9fd-491c-88e5-c0af7ab845d1" (UID: "782200a6-b9fd-491c-88e5-c0af7ab845d1"). InnerVolumeSpecName "kube-api-access-ksnn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.958583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.958562 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf" pod "914c666a-e17b-480d-8512-a90e7e8c165c" (UID: "914c666a-e17b-480d-8512-a90e7e8c165c"). InnerVolumeSpecName "kube-api-access-7c6hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.987157 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:21.986887 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:22.008757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.008730 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:22.056722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.056677 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6hf\" (UniqueName: \"kubernetes.io/projected/914c666a-e17b-480d-8512-a90e7e8c165c-kube-api-access-7c6hf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.057022 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.057004 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksnn6\" (UniqueName: \"kubernetes.io/projected/782200a6-b9fd-491c-88e5-c0af7ab845d1-kube-api-access-ksnn6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.057163 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.057150 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782200a6-b9fd-491c-88e5-c0af7ab845d1-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.057288 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.057274 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/914c666a-e17b-480d-8512-a90e7e8c165c-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.095178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.094212 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:22.158473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.158440 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\") pod \"0e3425fb-eeaf-474a-9200-7018ac566400\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " Oct 30 20:00:22.158659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.158640 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\" (UniqueName: \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\") pod \"0e3425fb-eeaf-474a-9200-7018ac566400\" (UID: \"0e3425fb-eeaf-474a-9200-7018ac566400\") " Oct 30 20:00:22.160319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.160170 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca" pod "0e3425fb-eeaf-474a-9200-7018ac566400" (UID: "0e3425fb-eeaf-474a-9200-7018ac566400"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:22.163994 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.163960 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f" pod "0e3425fb-eeaf-474a-9200-7018ac566400" (UID: "0e3425fb-eeaf-474a-9200-7018ac566400"). InnerVolumeSpecName "kube-api-access-mfn6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:22.173276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.173242 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:22.232717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.232689 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:22.239241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.239215 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:22.260392 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260365 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\" (UniqueName: \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\") pod \"1f599db9-6884-499a-b261-99ee7f7cd3a8\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " Oct 30 20:00:22.260618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260600 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\") pod \"1f599db9-6884-499a-b261-99ee7f7cd3a8\" (UID: \"1f599db9-6884-499a-b261-99ee7f7cd3a8\") " Oct 30 20:00:22.260685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260640 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\") pod \"596026b3-cdd7-4b4e-8416-d02c7b78364e\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " Oct 30 20:00:22.260750 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260699 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\" (UniqueName: \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\") pod \"596026b3-cdd7-4b4e-8416-d02c7b78364e\" (UID: \"596026b3-cdd7-4b4e-8416-d02c7b78364e\") " Oct 30 20:00:22.261359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260752 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\") pod \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " Oct 30 20:00:22.261359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.260845 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\" (UniqueName: \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\") pod \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\" (UID: \"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc\") " Oct 30 20:00:22.261359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.261057 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca" pod "596026b3-cdd7-4b4e-8416-d02c7b78364e" (UID: "596026b3-cdd7-4b4e-8416-d02c7b78364e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:22.261605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.261387 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3425fb-eeaf-474a-9200-7018ac566400-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.261605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.261412 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596026b3-cdd7-4b4e-8416-d02c7b78364e-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.261605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.261433 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfn6f\" (UniqueName: \"kubernetes.io/projected/0e3425fb-eeaf-474a-9200-7018ac566400-kube-api-access-mfn6f\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.261852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.261809 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca" pod "1f599db9-6884-499a-b261-99ee7f7cd3a8" (UID: "1f599db9-6884-499a-b261-99ee7f7cd3a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:22.262437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.262407 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca" pod "04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" (UID: "04cb01c5-4e7f-4cd5-8bd4-08c3148341cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:22.263565 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.263540 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw" pod "04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" (UID: "04cb01c5-4e7f-4cd5-8bd4-08c3148341cc"). InnerVolumeSpecName "kube-api-access-qz4qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:22.263650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.263628 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4" pod "1f599db9-6884-499a-b261-99ee7f7cd3a8" (UID: "1f599db9-6884-499a-b261-99ee7f7cd3a8"). InnerVolumeSpecName "kube-api-access-56jc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:22.263909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.263889 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79" pod "596026b3-cdd7-4b4e-8416-d02c7b78364e" (UID: "596026b3-cdd7-4b4e-8416-d02c7b78364e"). InnerVolumeSpecName "kube-api-access-6rv79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:22.362461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.362428 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4qw\" (UniqueName: \"kubernetes.io/projected/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-kube-api-access-qz4qw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.362461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.362456 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56jc4\" (UniqueName: \"kubernetes.io/projected/1f599db9-6884-499a-b261-99ee7f7cd3a8-kube-api-access-56jc4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.362461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.362465 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f599db9-6884-499a-b261-99ee7f7cd3a8-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.362683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.362478 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv79\" (UniqueName: \"kubernetes.io/projected/596026b3-cdd7-4b4e-8416-d02c7b78364e-kube-api-access-6rv79\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.362683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.362492 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:22.412846 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.412795 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" path="/var/lib/kubelet/pods/467b4713-0df1-4d0e-82bb-222aa22f7936/volumes" Oct 30 20:00:22.413605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.413576 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" path="/var/lib/kubelet/pods/ddf7c63a-b9c5-4532-8435-d61bcd804349/volumes" Oct 30 20:00:22.692810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.692783 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" Oct 30 20:00:22.693350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.692806 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt" event={"ID":"914c666a-e17b-480d-8512-a90e7e8c165c","Type":"ContainerDied","Data":"111383d07b81cb3d833d6b0b88dd7aab2f3b4a95b3f83622b43b04d0b78da55b"} Oct 30 20:00:22.693350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.692888 4990 scope.go:122] "RemoveContainer" containerID="c3bbe21b83331467a5b76433b3720a9ea9a809fc2d2d8df5fe1307378b543d44" Oct 30 20:00:22.694765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.694736 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" Oct 30 20:00:22.694892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.694778 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg" event={"ID":"1f599db9-6884-499a-b261-99ee7f7cd3a8","Type":"ContainerDied","Data":"a1f01c1e7470d498016f8a92f7268927c3b9ec36a9232ebcc4a08f05d0dfd2f7"} Oct 30 20:00:22.696775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.696739 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" Oct 30 20:00:22.696775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.696759 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb" event={"ID":"0e3425fb-eeaf-474a-9200-7018ac566400","Type":"ContainerDied","Data":"3de51ad214092a3806315b5f0ebef3a00d7c1b103dbaedb6ef852ccaa896955f"} Oct 30 20:00:22.699697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.699677 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" Oct 30 20:00:22.699852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.699713 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69" event={"ID":"04cb01c5-4e7f-4cd5-8bd4-08c3148341cc","Type":"ContainerDied","Data":"4152dd36506f89e0083eab5ecd6b81a081fabd9ede4bdebb873aafa32327c911"} Oct 30 20:00:22.701465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.701445 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" event={"ID":"782200a6-b9fd-491c-88e5-c0af7ab845d1","Type":"ContainerDied","Data":"0f467157c6d253f105c0341a33566e665daea927ac8e06f761a2e1dab7f5e105"} Oct 30 20:00:22.701629 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.701463 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm" Oct 30 20:00:22.703315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.703291 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" event={"ID":"596026b3-cdd7-4b4e-8416-d02c7b78364e","Type":"ContainerDied","Data":"51598ae37846e1d1165ce713358f5fb4a008f64685af4512091bde908e4a5e8e"} Oct 30 20:00:22.703315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.703309 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h" Oct 30 20:00:22.708217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.708201 4990 scope.go:122] "RemoveContainer" containerID="e0527e483f435d0f8242100fc06ad2a3cf9d76fc092e030823c65111c37cbda4" Oct 30 20:00:22.718183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.718160 4990 scope.go:122] "RemoveContainer" containerID="7131ce336c847f547a53db08dcd9e56c28717259e865cbfd456bf54a9866e641" Oct 30 20:00:22.729139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.729120 4990 scope.go:122] "RemoveContainer" containerID="99cbc2d34ff474e62332dcc55f49231f4ff277cf266e8c9cedefa4a411681cd2" Oct 30 20:00:22.739615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.739593 4990 scope.go:122] "RemoveContainer" containerID="17abace3f27d336a95b03fff65123c547e77ae5e333cbd3a67498406f020119f" Oct 30 20:00:22.741558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.741535 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:22.751165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.751007 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889840-crc69"] Oct 30 20:00:22.752567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.752549 4990 scope.go:122] "RemoveContainer" containerID="af8f72e34a49d79f3fb1b70736f72a1863e778e26690d1ec7d83b93128ca4594" Oct 30 20:00:22.781893 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.781868 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:22.796530 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.793019 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-mx55h"] Oct 30 20:00:22.822419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.822391 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:22.826336 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.826316 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e430a0-29889840-slwtg"] Oct 30 20:00:22.844568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.844523 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:22.874191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.873639 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:22.879293 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.879265 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889840-sh2dt"] Oct 30 20:00:22.904884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.904857 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:22.917557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.917529 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6e6016-29889840-qrmbm"] Oct 30 20:00:22.948446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.948388 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:22.952643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:22.952622 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cd4bde-29889840-cm9gb"] Oct 30 20:00:23.726235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.726204 4990 generic.go:356] "Generic (PLEG): container finished" podID="4f698579-c391-403d-923a-ff69ac928927" containerID="5e219ec191cdbafce96d7915da1fcb353f17ed81377827d6a978d5bfec68e37a" exitCode=0 Oct 30 20:00:23.726754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.726264 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" event={"ID":"4f698579-c391-403d-923a-ff69ac928927","Type":"ContainerDied","Data":"5e219ec191cdbafce96d7915da1fcb353f17ed81377827d6a978d5bfec68e37a"} Oct 30 20:00:23.807157 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.807128 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 20:00:23.895301 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895245 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895320 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895364 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895459 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895535 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895576 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895623 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895662 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:23.895814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895736 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.895919 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.895833 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\") pod \"4f698579-c391-403d-923a-ff69ac928927\" (UID: \"4f698579-c391-403d-923a-ff69ac928927\") " Oct 30 20:00:23.896448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.896415 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:23.896608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.896565 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.896679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.896629 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f698579-c391-403d-923a-ff69ac928927-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.896679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.896644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:23.896848 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.896817 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:23.897022 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.897002 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:23.901517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.901482 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:23.901796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.901615 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:23.902295 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.902267 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "kube-api-access-n4z5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:23.905446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.905416 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token" pod "4f698579-c391-403d-923a-ff69ac928927" (UID: "4f698579-c391-403d-923a-ff69ac928927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:23.997522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997494 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f698579-c391-403d-923a-ff69ac928927-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997521 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4z5x\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-kube-api-access-n4z5x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997531 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997540 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f698579-c391-403d-923a-ff69ac928927-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997548 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997555 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:23.997707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:23.997564 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f698579-c391-403d-923a-ff69ac928927-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:24.408212 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.408179 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" path="/var/lib/kubelet/pods/04cb01c5-4e7f-4cd5-8bd4-08c3148341cc/volumes" Oct 30 20:00:24.408653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.408637 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" path="/var/lib/kubelet/pods/0e3425fb-eeaf-474a-9200-7018ac566400/volumes" Oct 30 20:00:24.409074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.409058 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" path="/var/lib/kubelet/pods/1f599db9-6884-499a-b261-99ee7f7cd3a8/volumes" Oct 30 20:00:24.409810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.409792 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" path="/var/lib/kubelet/pods/596026b3-cdd7-4b4e-8416-d02c7b78364e/volumes" Oct 30 20:00:24.410244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.410229 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" path="/var/lib/kubelet/pods/782200a6-b9fd-491c-88e5-c0af7ab845d1/volumes" Oct 30 20:00:24.410633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.410620 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" path="/var/lib/kubelet/pods/914c666a-e17b-480d-8512-a90e7e8c165c/volumes" Oct 30 20:00:24.739456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.739376 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" Oct 30 20:00:24.739456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.739416 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz" event={"ID":"4f698579-c391-403d-923a-ff69ac928927","Type":"ContainerDied","Data":"7c9cab264b5c88486cf72baf2e707b3baa66f4a07202db246847de93c72e7317"} Oct 30 20:00:24.739935 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.739480 4990 scope.go:122] "RemoveContainer" containerID="5e219ec191cdbafce96d7915da1fcb353f17ed81377827d6a978d5bfec68e37a" Oct 30 20:00:24.795824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.795777 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 20:00:24.805726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:24.805697 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-mdhxz"] Oct 30 20:00:25.024037 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:25.023943 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.403454 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.403415 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f698579-c391-403d-923a-ff69ac928927" path="/var/lib/kubelet/pods/4f698579-c391-403d-923a-ff69ac928927/volumes" Oct 30 20:00:26.440466 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.440434 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.596815 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.596774 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.733689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.733604 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 20:00:26.733890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.733858 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" containerID="cri-o://8d8d15b5934c13c4d37f9a9eca5b9b1e628d802cb2a2a10d884ea4bdcd028232" gracePeriod=30 Oct 30 20:00:26.816388 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.816354 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.964002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:26.963964 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:27.236232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236193 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd"] Oct 30 20:00:27.236925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236911 4990 cpu_manager.go:414] "containerMap: removing container" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236927 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236938 4990 cpu_manager.go:414] "containerMap: removing container" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236945 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236957 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236962 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236971 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.236974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236976 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236983 4990 cpu_manager.go:414] "containerMap: removing container" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236989 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.236999 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237003 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237015 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237020 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237029 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237033 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237042 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237047 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237055 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ff27e004-cb42-4336-81cb-9881e663796b" containerName="payload-pre-processing" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237059 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff27e004-cb42-4336-81cb-9881e663796b" containerName="payload-pre-processing" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237066 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237071 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237078 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237082 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237106 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237111 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237117 4990 cpu_manager.go:414] "containerMap: removing container" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237122 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237131 4990 cpu_manager.go:414] "containerMap: removing container" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237136 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237144 4990 cpu_manager.go:414] "containerMap: removing container" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237149 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237159 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237163 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237168 4990 cpu_manager.go:414] "containerMap: removing container" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237173 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237179 4990 cpu_manager.go:414] "containerMap: removing container" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237184 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237190 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237195 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237203 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237207 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237217 4990 cpu_manager.go:414] "containerMap: removing container" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237221 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.237214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237228 4990 cpu_manager.go:414] "containerMap: removing container" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237233 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237241 4990 cpu_manager.go:414] "containerMap: removing container" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237245 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237251 4990 cpu_manager.go:414] "containerMap: removing container" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237255 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237261 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237265 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237270 4990 cpu_manager.go:414] "containerMap: removing container" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237275 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237280 4990 cpu_manager.go:414] "containerMap: removing container" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237285 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237290 4990 cpu_manager.go:414] "containerMap: removing container" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237294 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237302 4990 cpu_manager.go:414] "containerMap: removing container" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237306 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237313 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237317 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237418 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237425 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237430 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237435 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237442 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f698579-c391-403d-923a-ff69ac928927" containerName="istio-proxy" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237449 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237454 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237460 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237466 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237471 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237476 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237482 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237487 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237491 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5468dfab-1958-4793-a2e3-f83df3930d12" containerName="payload-processing" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237498 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237504 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237511 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237517 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237522 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237527 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237533 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237539 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1564f703-b9e7-4851-8ddd-e1f6b7b0a401" containerName="payload-pre-processing" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237545 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237552 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff27e004-cb42-4336-81cb-9881e663796b" containerName="payload-pre-processing" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237558 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237564 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237571 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3425fb-eeaf-474a-9200-7018ac566400" containerName="cleanup" Oct 30 20:00:27.238419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.237577 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:27.241336 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.241317 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.244418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.244392 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.244558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.244535 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.244619 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.244593 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drz62\" (UniqueName: \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.258713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.258689 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd"] Oct 30 20:00:27.272060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.272038 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:27.272704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.272464 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:00:27.345770 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.345745 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.345892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.345829 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.345892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.345859 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drz62\" (UniqueName: \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.346426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.346401 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.348250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.348224 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.363695 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.363671 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drz62\" (UniqueName: \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\") pod \"maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.555930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.555906 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:27.868900 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:27.868864 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd"] Oct 30 20:00:27.869045 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:27.869023 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92406b00_b6c5_4618_b930_92720c5ff0b3.slice/crio-51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13 WatchSource:0}: Error finding container 51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13: Status 404 returned error can't find the container with id 51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13 Oct 30 20:00:28.725213 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:28.725170 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.782384 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:28.782348 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" event={"ID":"92406b00-b6c5-4618-b930-92720c5ff0b3","Type":"ContainerStarted","Data":"1358b7d6f79a7aed170aaf1dd7b318b70416a6b32b5ab64decb60e25dc51097c"} Oct 30 20:00:28.782558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:28.782413 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" event={"ID":"92406b00-b6c5-4618-b930-92720c5ff0b3","Type":"ContainerStarted","Data":"51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13"} Oct 30 20:00:28.821172 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:28.821119 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" podStartSLOduration=1.821075351 podStartE2EDuration="1.821075351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:28.818216765 +0000 UTC m=+3336.529742146" watchObservedRunningTime="2026-10-30 20:00:28.821075351 +0000 UTC m=+3336.532600754" Oct 30 20:00:28.898146 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:28.895841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.133235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.133202 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 20:00:29.169039 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.169006 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 20:00:29.326647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.326609 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.484336 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.484239 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.935117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.928146 4990 generic.go:356] "Generic (PLEG): container finished" podID="92406b00-b6c5-4618-b930-92720c5ff0b3" containerID="1358b7d6f79a7aed170aaf1dd7b318b70416a6b32b5ab64decb60e25dc51097c" exitCode=0 Oct 30 20:00:29.935117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:29.928267 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" event={"ID":"92406b00-b6c5-4618-b930-92720c5ff0b3","Type":"ContainerDied","Data":"1358b7d6f79a7aed170aaf1dd7b318b70416a6b32b5ab64decb60e25dc51097c"} Oct 30 20:00:30.367443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.367405 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.417626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.417589 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 20:00:30.417832 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.417806 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" podUID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" containerName="payload-processing" containerID="cri-o://8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391" gracePeriod=30 Oct 30 20:00:30.548823 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.548781 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.691754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.683886 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 20:00:30.691754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.684137 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" podUID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" containerName="maas-api" containerID="cri-o://a2bffee85c928f30402bd6786def685682a79b255a5e17c831bc590f0d93346e" gracePeriod=30 Oct 30 20:00:30.714836 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.714729 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.850340 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.850310 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 20:00:30.870173 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.870144 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.950631 4990 generic.go:356] "Generic (PLEG): container finished" podID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" containerID="8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391" exitCode=0 Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.950717 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" event={"ID":"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff","Type":"ContainerDied","Data":"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391"} Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.950765 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" event={"ID":"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff","Type":"ContainerDied","Data":"89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538"} Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.950799 4990 scope.go:122] "RemoveContainer" containerID="8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391" Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.950957 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v" Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.955116 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\") pod \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.955321 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\" (UniqueName: \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\") pod \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\" (UID: \"0c4247cd-4e68-4443-8fe8-3f10e59ae5ff\") " Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.956824 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume" pod "0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" (UID: "0c4247cd-4e68-4443-8fe8-3f10e59ae5ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.959610 4990 generic.go:356] "Generic (PLEG): container finished" podID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" containerID="a2bffee85c928f30402bd6786def685682a79b255a5e17c831bc590f0d93346e" exitCode=0 Oct 30 20:00:30.960124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.959953 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" event={"ID":"9ed5c899-84c4-4eb6-b6e7-9c716822b923","Type":"ContainerDied","Data":"a2bffee85c928f30402bd6786def685682a79b255a5e17c831bc590f0d93346e"} Oct 30 20:00:31.032206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:30.967826 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w" pod "0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" (UID: "0c4247cd-4e68-4443-8fe8-3f10e59ae5ff"). InnerVolumeSpecName "kube-api-access-6br6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:31.032206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.029395 4990 scope.go:122] "RemoveContainer" containerID="8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391" Oct 30 20:00:31.033015 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:31.032774 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391\": container with ID starting with 8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391 not found: ID does not exist" containerID="8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391" Oct 30 20:00:31.033015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.032811 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391"} err="failed to get container status \"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391\": rpc error: code = NotFound desc = could not find container \"8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391\": container with ID starting with 8cb505ce77b0ec106b600b9283a06d67a13b4ed73faae11104784fe827641391 not found: ID does not exist" Oct 30 20:00:31.035490 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.034139 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.041621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.041548 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz"] Oct 30 20:00:31.094395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.061630 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6br6w\" (UniqueName: \"kubernetes.io/projected/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-kube-api-access-6br6w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.094395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.061665 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.094395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.081834 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 20:00:31.135318 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.135286 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ed2ede-ldrnz"] Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.214587 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\") pod \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.214671 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\" (UniqueName: \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\") pod \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.214896 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\") pod \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.214949 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\") pod \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\" (UID: \"9ed5c899-84c4-4eb6-b6e7-9c716822b923\") " Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.219409 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca" pod "9ed5c899-84c4-4eb6-b6e7-9c716822b923" (UID: "9ed5c899-84c4-4eb6-b6e7-9c716822b923"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.219875 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw" pod "9ed5c899-84c4-4eb6-b6e7-9c716822b923" (UID: "9ed5c899-84c4-4eb6-b6e7-9c716822b923"). InnerVolumeSpecName "kube-api-access-hzdjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.222122 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls" pod "9ed5c899-84c4-4eb6-b6e7-9c716822b923" (UID: "9ed5c899-84c4-4eb6-b6e7-9c716822b923"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:31.225161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.222136 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs" pod "9ed5c899-84c4-4eb6-b6e7-9c716822b923" (UID: "9ed5c899-84c4-4eb6-b6e7-9c716822b923"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:31.316757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.316725 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed5c899-84c4-4eb6-b6e7-9c716822b923-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.316757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.316757 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.316948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.316773 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ed5c899-84c4-4eb6-b6e7-9c716822b923-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.316948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.316786 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdjw\" (UniqueName: \"kubernetes.io/projected/9ed5c899-84c4-4eb6-b6e7-9c716822b923-kube-api-access-hzdjw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.328877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.328849 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 20:00:31.346692 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.346661 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ed2ede-6ff5969f6d-v226v"] Oct 30 20:00:31.364530 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:31.364502 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4247cd_4e68_4443_8fe8_3f10e59ae5ff.slice/crio-89166a0d282b9e60b890a3a73b7e72f3f89d4083b1492423f78be30757f2b538\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4247cd_4e68_4443_8fe8_3f10e59ae5ff.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:00:31.398124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.398085 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:31.518906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.518833 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\") pod \"92406b00-b6c5-4618-b930-92720c5ff0b3\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " Oct 30 20:00:31.518906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.518901 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\" (UniqueName: \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\") pod \"92406b00-b6c5-4618-b930-92720c5ff0b3\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " Oct 30 20:00:31.519166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.519048 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\") pod \"92406b00-b6c5-4618-b930-92720c5ff0b3\" (UID: \"92406b00-b6c5-4618-b930-92720c5ff0b3\") " Oct 30 20:00:31.519369 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.519310 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca" pod "92406b00-b6c5-4618-b930-92720c5ff0b3" (UID: "92406b00-b6c5-4618-b930-92720c5ff0b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:31.519765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.519740 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.521661 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.521636 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca" pod "92406b00-b6c5-4618-b930-92720c5ff0b3" (UID: "92406b00-b6c5-4618-b930-92720c5ff0b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:31.521783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.521658 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62" pod "92406b00-b6c5-4618-b930-92720c5ff0b3" (UID: "92406b00-b6c5-4618-b930-92720c5ff0b3"). InnerVolumeSpecName "kube-api-access-drz62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:31.587244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.587209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.621205 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.621177 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drz62\" (UniqueName: \"kubernetes.io/projected/92406b00-b6c5-4618-b930-92720c5ff0b3-kube-api-access-drz62\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.621205 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.621203 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92406b00-b6c5-4618-b930-92720c5ff0b3-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.738897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.738859 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.887745 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.887713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.971922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.971883 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" event={"ID":"92406b00-b6c5-4618-b930-92720c5ff0b3","Type":"ContainerDied","Data":"51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13"} Oct 30 20:00:31.972351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.971980 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a72ac494df49e400949b048167e81781ab4a32265de001f46973b994f8be13" Oct 30 20:00:31.972351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.972061 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd" Oct 30 20:00:31.983775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.983395 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" event={"ID":"9ed5c899-84c4-4eb6-b6e7-9c716822b923","Type":"ContainerDied","Data":"fa1048e4bc8578c2b77dca83261a512f5c36f507c8827b202e2f2f092ffe5a6b"} Oct 30 20:00:31.983775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.983449 4990 scope.go:122] "RemoveContainer" containerID="a2bffee85c928f30402bd6786def685682a79b255a5e17c831bc590f0d93346e" Oct 30 20:00:31.983775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.983578 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4" Oct 30 20:00:31.993142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.993076 4990 generic.go:356] "Generic (PLEG): container finished" podID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerID="8d8d15b5934c13c4d37f9a9eca5b9b1e628d802cb2a2a10d884ea4bdcd028232" exitCode=0 Oct 30 20:00:31.993314 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:31.993240 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" event={"ID":"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351","Type":"ContainerDied","Data":"8d8d15b5934c13c4d37f9a9eca5b9b1e628d802cb2a2a10d884ea4bdcd028232"} Oct 30 20:00:32.021977 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.021955 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 20:00:32.063274 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.063186 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 20:00:32.067514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.067487 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-f83282-79f9589896-knld4"] Oct 30 20:00:32.131889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.131856 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.131972 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132122 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132187 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132305 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132235 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132305 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132286 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132325 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132459 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132426 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.132517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.132489 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\") pod \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\" (UID: \"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351\") " Oct 30 20:00:32.133428 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.133395 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.134061 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.134029 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:32.134399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.134373 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.134583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.134562 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.134833 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.134812 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.135157 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.135126 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.136645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.136617 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "kube-api-access-z8lxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.136786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.136761 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:32.137308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.137287 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy" pod "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" (UID: "4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.234916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234882 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.234916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234920 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234932 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234947 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234956 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234966 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234975 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lxt\" (UniqueName: \"kubernetes.io/projected/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-kube-api-access-z8lxt\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234983 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.234993 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.235871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.235837 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.404727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.404643 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" path="/var/lib/kubelet/pods/0c4247cd-4e68-4443-8fe8-3f10e59ae5ff/volumes" Oct 30 20:00:32.405231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.405208 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b128b2d-bca1-45e8-aaaa-7608dca8a1fc" path="/var/lib/kubelet/pods/5b128b2d-bca1-45e8-aaaa-7608dca8a1fc/volumes" Oct 30 20:00:32.405693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:32.405673 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" path="/var/lib/kubelet/pods/9ed5c899-84c4-4eb6-b6e7-9c716822b923/volumes" Oct 30 20:00:33.006295 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.006264 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" Oct 30 20:00:33.006818 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.006299 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb" event={"ID":"4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351","Type":"ContainerDied","Data":"18bb52732b7adbee25492f62460e303521e28f4dd6d3d13d44769fe9e2c7d8a6"} Oct 30 20:00:33.006818 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.006362 4990 scope.go:122] "RemoveContainer" containerID="8d8d15b5934c13c4d37f9a9eca5b9b1e628d802cb2a2a10d884ea4bdcd028232" Oct 30 20:00:33.069017 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.068984 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 20:00:33.073286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.073261 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-h9dqb"] Oct 30 20:00:33.524570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:33.524382 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:34.133511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:34.133469 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" containerID="cri-o://42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740" gracePeriod=25 Oct 30 20:00:34.170127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:34.170048 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" containerID="cri-o://d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949" gracePeriod=25 Oct 30 20:00:34.405637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:34.405531 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" path="/var/lib/kubelet/pods/4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351/volumes" Oct 30 20:00:35.161241 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:35.161200 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:00:35.337237 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:35.337194 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.474815 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:36.474776 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 20:00:36.590376 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:36.590340 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.990714 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:36.990679 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.273285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.273236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.425960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.425920 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.522042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.522010 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 20:00:38.522274 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.522246 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" containerID="cri-o://191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4" gracePeriod=30 Oct 30 20:00:38.600352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.600321 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.806924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.806807 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.927956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.927880 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 20:00:38.928139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:38.928078 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" containerID="cri-o://a4aad552597dee35ca2a94c30c2d30a076f11fc985f1a9e378459ebd7b889f36" gracePeriod=30 Oct 30 20:00:39.142345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.142312 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.201156 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.201049 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4"] Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202376 4990 cpu_manager.go:414] "containerMap: removing container" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202403 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467b4713-0df1-4d0e-82bb-222aa22f7936" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202431 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202439 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202470 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202478 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c70006-7ab0-4e77-bd20-3689b18ebdb5" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202518 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" containerName="payload-processing" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202534 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" containerName="payload-processing" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202566 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" containerName="maas-api" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202573 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" containerName="maas-api" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202612 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202619 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202650 4990 cpu_manager.go:414] "containerMap: removing container" podUID="92406b00-b6c5-4618-b930-92720c5ff0b3" containerName="revoke-keys" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202658 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92406b00-b6c5-4618-b930-92720c5ff0b3" containerName="revoke-keys" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202705 4990 cpu_manager.go:414] "containerMap: removing container" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202713 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202775 4990 cpu_manager.go:414] "containerMap: removing container" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.202783 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203407 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="596026b3-cdd7-4b4e-8416-d02c7b78364e" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203428 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf7c63a-b9c5-4532-8435-d61bcd804349" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203456 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="914c666a-e17b-480d-8512-a90e7e8c165c" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203473 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cb01c5-4e7f-4cd5-8bd4-08c3148341cc" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203497 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0ac18d-5b71-4bc1-8ec4-5d92ded4f351" containerName="istio-proxy" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203513 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="782200a6-b9fd-491c-88e5-c0af7ab845d1" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203533 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f599db9-6884-499a-b261-99ee7f7cd3a8" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203544 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="62372dcf-ea83-4c7a-a3f9-96007c3aafcb" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203562 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d9402f-122c-4206-a6e9-0ceda82c6117" containerName="cleanup" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203578 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4247cd-4e68-4443-8fe8-3f10e59ae5ff" containerName="payload-processing" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203590 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed5c899-84c4-4eb6-b6e7-9c716822b923" containerName="maas-api" Oct 30 20:00:39.203700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.203614 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="92406b00-b6c5-4618-b930-92720c5ff0b3" containerName="revoke-keys" Oct 30 20:00:39.216734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.216706 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.224487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.224460 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4"] Oct 30 20:00:39.229927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.229903 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:00:39.230122 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.229920 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:39.257954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.257919 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.258117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.258019 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgdf\" (UniqueName: \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.258117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.258063 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.359016 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.358986 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.359496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.359087 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgdf\" (UniqueName: \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.359496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.359187 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.359787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.359763 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.361506 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.361483 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.376909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.376886 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgdf\" (UniqueName: \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.533816 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.533737 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:39.844856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:39.844820 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4"] Oct 30 20:00:39.846698 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:39.846670 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edc69a9_dfb0_4664_ab2a_fc16f31cf08d.slice/crio-282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3 WatchSource:0}: Error finding container 282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3: Status 404 returned error can't find the container with id 282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3 Oct 30 20:00:40.074653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:40.074615 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" event={"ID":"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d","Type":"ContainerStarted","Data":"282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3"} Oct 30 20:00:41.022786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.022758 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:41.023229 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.022756 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:41.027740 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.027717 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn"] Oct 30 20:00:41.037508 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.037487 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693a1b3c-dgkqn"] Oct 30 20:00:41.086455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.086423 4990 generic.go:356] "Generic (PLEG): container finished" podID="6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" containerID="5cf8a0cfe05f60edf0d8f1c2b3721a5279b53ee464791332e0278cec396fa329" exitCode=0 Oct 30 20:00:41.086596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.086509 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" event={"ID":"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d","Type":"ContainerDied","Data":"5cf8a0cfe05f60edf0d8f1c2b3721a5279b53ee464791332e0278cec396fa329"} Oct 30 20:00:41.164385 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:41.161639 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.269805 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.269776 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:42.299292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.299264 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\") pod \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " Oct 30 20:00:42.299423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.299401 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\") pod \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " Oct 30 20:00:42.299628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.299605 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca" pod "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" (UID: "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:42.299666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.299628 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\" (UniqueName: \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\") pod \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\" (UID: \"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d\") " Oct 30 20:00:42.300283 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.300255 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:42.301556 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.301524 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca" pod "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" (UID: "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:42.301675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.301655 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf" pod "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" (UID: "6edc69a9-dfb0-4664-ab2a-fc16f31cf08d"). InnerVolumeSpecName "kube-api-access-6jgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:42.401765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.401690 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:42.401765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.401719 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgdf\" (UniqueName: \"kubernetes.io/projected/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d-kube-api-access-6jgdf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:42.405435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.405408 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b78e2c-855a-49e2-9de8-4fc5a8cfe984" path="/var/lib/kubelet/pods/65b78e2c-855a-49e2-9de8-4fc5a8cfe984/volumes" Oct 30 20:00:42.914177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:42.914131 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.112386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.112359 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" Oct 30 20:00:43.112543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.112401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4" event={"ID":"6edc69a9-dfb0-4664-ab2a-fc16f31cf08d","Type":"ContainerDied","Data":"282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3"} Oct 30 20:00:43.112543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.112461 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="282936af8ac7f531133feed3042ee5f310cb2dea575ff96e37ca108b707387b3" Oct 30 20:00:43.831734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.831702 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 20:00:43.931040 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931006 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931053 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931080 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931126 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931205 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931315 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931324 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.931494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931337 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931416 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931439 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.931753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931511 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.931753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931522 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\") pod \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\" (UID: \"3ff90ce6-b0c6-4237-8714-36b0986e19ce\") " Oct 30 20:00:43.931753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931544 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:43.931906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.931832 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.932436 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.932407 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.932557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.932440 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.932557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.932453 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.932557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.932464 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.932557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.932477 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.934227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.934195 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:43.934511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.934486 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:43.934610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.934518 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "kube-api-access-jhmp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:43.935127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:43.935111 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy" pod "3ff90ce6-b0c6-4237-8714-36b0986e19ce" (UID: "3ff90ce6-b0c6-4237-8714-36b0986e19ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.033681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.033639 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmp8\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-kube-api-access-jhmp8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.033681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.033679 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.033876 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.033696 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.033876 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.033709 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff90ce6-b0c6-4237-8714-36b0986e19ce-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.125475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.125439 4990 generic.go:356] "Generic (PLEG): container finished" podID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerID="191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4" exitCode=0 Oct 30 20:00:44.125837 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.125658 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" event={"ID":"3ff90ce6-b0c6-4237-8714-36b0986e19ce","Type":"ContainerDied","Data":"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4"} Oct 30 20:00:44.125973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.125952 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" event={"ID":"3ff90ce6-b0c6-4237-8714-36b0986e19ce","Type":"ContainerDied","Data":"d8ccd8f629241d372e2a4d8b299b5a399951f653e8ab777a0a869d61174a614a"} Oct 30 20:00:44.126050 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.125999 4990 scope.go:122] "RemoveContainer" containerID="191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4" Oct 30 20:00:44.126175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.125695 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4" Oct 30 20:00:44.130227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.128688 4990 generic.go:356] "Generic (PLEG): container finished" podID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerID="a4aad552597dee35ca2a94c30c2d30a076f11fc985f1a9e378459ebd7b889f36" exitCode=0 Oct 30 20:00:44.130227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.128900 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" event={"ID":"523c919e-98f3-4b9d-8199-73d2d9ae649e","Type":"ContainerDied","Data":"a4aad552597dee35ca2a94c30c2d30a076f11fc985f1a9e378459ebd7b889f36"} Oct 30 20:00:44.145612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.145587 4990 scope.go:122] "RemoveContainer" containerID="191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4" Oct 30 20:00:44.146144 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:44.146066 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4\": container with ID starting with 191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4 not found: ID does not exist" containerID="191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4" Oct 30 20:00:44.146144 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.146117 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4"} err="failed to get container status \"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4\": rpc error: code = NotFound desc = could not find container \"191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4\": container with ID starting with 191c081bdba943ae46a4074e0a111171ae0598297f04009e3b5c89c140bb0bf4 not found: ID does not exist" Oct 30 20:00:44.156330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.156298 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.178921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.178889 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 20:00:44.186482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.186445 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-5sgh4"] Oct 30 20:00:44.290191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.286477 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 20:00:44.317956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.317868 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.337525 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337488 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.337709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337632 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.337709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337689 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.337831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337714 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.337831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337750 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.337988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.337961 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.338153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338014 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.338153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338080 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.338153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338143 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.338356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338172 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.338356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338216 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\") pod \"523c919e-98f3-4b9d-8199-73d2d9ae649e\" (UID: \"523c919e-98f3-4b9d-8199-73d2d9ae649e\") " Oct 30 20:00:44.338450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338425 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.338639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338613 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:44.338734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.338712 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.339202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.339177 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.339323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.339206 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.339323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.339223 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.339323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.339237 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523c919e-98f3-4b9d-8199-73d2d9ae649e-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.339323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.339251 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.340434 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.340402 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "kube-api-access-nft9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:44.340603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.340582 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:44.340738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.340716 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:44.342407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.342384 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy" pod "523c919e-98f3-4b9d-8199-73d2d9ae649e" (UID: "523c919e-98f3-4b9d-8199-73d2d9ae649e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:44.403920 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.403885 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" path="/var/lib/kubelet/pods/3ff90ce6-b0c6-4237-8714-36b0986e19ce/volumes" Oct 30 20:00:44.441768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.441738 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.441768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.441772 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.441990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.441808 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.441990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.441832 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nft9t\" (UniqueName: \"kubernetes.io/projected/523c919e-98f3-4b9d-8199-73d2d9ae649e-kube-api-access-nft9t\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.527363 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528539 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528561 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528585 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" containerName="revoke-keys" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528592 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" containerName="revoke-keys" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528612 4990 cpu_manager.go:414] "containerMap: removing container" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528620 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528755 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff90ce6-b0c6-4237-8714-36b0986e19ce" containerName="istio-proxy" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528771 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" containerName="revoke-keys" Oct 30 20:00:44.534452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.528782 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" containerName="istio-proxy" Oct 30 20:00:44.541112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.539431 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.577276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.559616 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e430a0-dockercfg-7cbdf\"" Oct 30 20:00:44.577276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.560307 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e430a0\"" Oct 30 20:00:44.577276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.575985 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:00:44.678105 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.678058 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq8n\" (UniqueName: \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.678294 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.678277 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.696360 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.696329 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:00:44.700692 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.700672 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.720328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.720304 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:00:44.779406 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.779377 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq8n\" (UniqueName: \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.779572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.779476 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdclc\" (UniqueName: \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.779572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.779523 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.779677 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.779661 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.780595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.780571 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.797937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.797909 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq8n\" (UniqueName: \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:44.881548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.881477 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdclc\" (UniqueName: \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.881548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.881526 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.882277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.882257 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.899391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.899361 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdclc\" (UniqueName: \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\") pod \"payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:44.919641 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:44.919611 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:45.015666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.015624 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:45.146895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.146815 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" event={"ID":"523c919e-98f3-4b9d-8199-73d2d9ae649e","Type":"ContainerDied","Data":"db9ef7b7addb96630d16eec7a7b0163cb60640a0bcce2c3b2b6bfebbfab42e6d"} Oct 30 20:00:45.146895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.146834 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml" Oct 30 20:00:45.146895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.146881 4990 scope.go:122] "RemoveContainer" containerID="a4aad552597dee35ca2a94c30c2d30a076f11fc985f1a9e378459ebd7b889f36" Oct 30 20:00:45.198912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.198884 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 20:00:45.204201 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.204172 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-blpml"] Oct 30 20:00:45.235108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.235012 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.240728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.240701 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:00:45.241651 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:45.241574 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0bf653_b968_4b38_a502_491d4a4c1a59.slice/crio-2a95838c86a157a49652af3d59acc21d7f91afc6dcf590bfcec6e23f72612937 WatchSource:0}: Error finding container 2a95838c86a157a49652af3d59acc21d7f91afc6dcf590bfcec6e23f72612937: Status 404 returned error can't find the container with id 2a95838c86a157a49652af3d59acc21d7f91afc6dcf590bfcec6e23f72612937 Oct 30 20:00:45.333313 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:45.333287 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:00:45.334080 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:00:45.334056 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c4764_992b_445c_8efd_6198b7243572.slice/crio-ca549f94e5b14320f52151568f9041552becb29587785c6665625794726ef821 WatchSource:0}: Error finding container ca549f94e5b14320f52151568f9041552becb29587785c6665625794726ef821: Status 404 returned error can't find the container with id ca549f94e5b14320f52151568f9041552becb29587785c6665625794726ef821 Oct 30 20:00:46.159135 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.159082 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" event={"ID":"845c4764-992b-445c-8efd-6198b7243572","Type":"ContainerStarted","Data":"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667"} Oct 30 20:00:46.159593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.159161 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" event={"ID":"845c4764-992b-445c-8efd-6198b7243572","Type":"ContainerStarted","Data":"ca549f94e5b14320f52151568f9041552becb29587785c6665625794726ef821"} Oct 30 20:00:46.159593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.159300 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:46.161946 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.161919 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" event={"ID":"1c0bf653-b968-4b38-a502-491d4a4c1a59","Type":"ContainerStarted","Data":"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a"} Oct 30 20:00:46.162075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.161967 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" event={"ID":"1c0bf653-b968-4b38-a502-491d4a4c1a59","Type":"ContainerStarted","Data":"2a95838c86a157a49652af3d59acc21d7f91afc6dcf590bfcec6e23f72612937"} Oct 30 20:00:46.162075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.162005 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:46.199590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.199539 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" podStartSLOduration=2.199525183 podStartE2EDuration="2.199525183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:46.194922482 +0000 UTC m=+3353.906447848" watchObservedRunningTime="2026-10-30 20:00:46.199525183 +0000 UTC m=+3353.911050549" Oct 30 20:00:46.233909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.233855 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" podStartSLOduration=2.233830397 podStartE2EDuration="2.233830397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:46.228836384 +0000 UTC m=+3353.940361750" watchObservedRunningTime="2026-10-30 20:00:46.233830397 +0000 UTC m=+3353.945355765" Oct 30 20:00:46.402665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.402629 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523c919e-98f3-4b9d-8199-73d2d9ae649e" path="/var/lib/kubelet/pods/523c919e-98f3-4b9d-8199-73d2d9ae649e/volumes" Oct 30 20:00:46.477238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.477137 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 20:00:46.477419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.477376 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" podUID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" containerName="maas-api" containerID="cri-o://137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43" gracePeriod=30 Oct 30 20:00:46.584400 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.584362 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.785204 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.785172 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 20:00:46.788475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.788444 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.917597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.917561 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\") pod \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " Oct 30 20:00:46.917808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.917631 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\") pod \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " Oct 30 20:00:46.917808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.917799 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\") pod \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " Oct 30 20:00:46.917943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.917831 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\" (UniqueName: \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\") pod \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\" (UID: \"d456f4c8-a2f1-4ad3-8f6e-19ac98065606\") " Oct 30 20:00:46.918057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.918029 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca" pod "d456f4c8-a2f1-4ad3-8f6e-19ac98065606" (UID: "d456f4c8-a2f1-4ad3-8f6e-19ac98065606"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:46.918461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.918440 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:46.920262 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.920221 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs" pod "d456f4c8-a2f1-4ad3-8f6e-19ac98065606" (UID: "d456f4c8-a2f1-4ad3-8f6e-19ac98065606"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:46.920768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.920741 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls" pod "d456f4c8-a2f1-4ad3-8f6e-19ac98065606" (UID: "d456f4c8-a2f1-4ad3-8f6e-19ac98065606"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:46.920871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.920761 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj" pod "d456f4c8-a2f1-4ad3-8f6e-19ac98065606" (UID: "d456f4c8-a2f1-4ad3-8f6e-19ac98065606"). InnerVolumeSpecName "kube-api-access-hhnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:46.934675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:46.934643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.027435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.027330 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.027435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.027366 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.027435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.027382 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhnpj\" (UniqueName: \"kubernetes.io/projected/d456f4c8-a2f1-4ad3-8f6e-19ac98065606-kube-api-access-hhnpj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.175843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.175800 4990 generic.go:356] "Generic (PLEG): container finished" podID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" containerID="137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43" exitCode=0 Oct 30 20:00:47.175843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.175839 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" Oct 30 20:00:47.176411 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.175894 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" event={"ID":"d456f4c8-a2f1-4ad3-8f6e-19ac98065606","Type":"ContainerDied","Data":"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43"} Oct 30 20:00:47.176411 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.175971 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk" event={"ID":"d456f4c8-a2f1-4ad3-8f6e-19ac98065606","Type":"ContainerDied","Data":"c1ad22e4a0c406dde6cc100a9e5248ecad34c07304ad4265a06832f3c5b20699"} Oct 30 20:00:47.176411 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.176186 4990 scope.go:122] "RemoveContainer" containerID="137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43" Oct 30 20:00:47.198712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.198689 4990 scope.go:122] "RemoveContainer" containerID="137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43" Oct 30 20:00:47.202586 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:47.202547 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43\": container with ID starting with 137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43 not found: ID does not exist" containerID="137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43" Oct 30 20:00:47.202698 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.202592 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43"} err="failed to get container status \"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43\": rpc error: code = NotFound desc = could not find container \"137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43\": container with ID starting with 137088c79c72b3a453565e26c8072bb1ba3e2e6f9991083f7fa89f54f51d9e43 not found: ID does not exist" Oct 30 20:00:47.241500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.241468 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 20:00:47.248476 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.248449 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e6016-97dd97c6b-zrcnk"] Oct 30 20:00:47.323906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.323876 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.443816 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.443782 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd"] Oct 30 20:00:47.447524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.447501 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6e6016-mfrsd"] Oct 30 20:00:47.479602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.479569 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.630165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.630062 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.954984 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:47.954889 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.401998 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:48.401966 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92406b00-b6c5-4618-b930-92720c5ff0b3" path="/var/lib/kubelet/pods/92406b00-b6c5-4618-b930-92720c5ff0b3/volumes" Oct 30 20:00:48.402529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:48.402513 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" path="/var/lib/kubelet/pods/d456f4c8-a2f1-4ad3-8f6e-19ac98065606/volumes" Oct 30 20:00:49.505729 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:49.505627 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.754667 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:49.754628 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 20:00:49.910464 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:49.910434 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:50.066635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:50.065951 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.022884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.022848 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:51.022884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.022874 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:51.290593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.290494 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.496738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.496689 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" podUID="09f6563c-411e-4018-b4e0-531f774367f0" containerName="main" containerID="cri-o://02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e" gracePeriod=45 Oct 30 20:00:51.780058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.780033 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 20:00:51.810083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810052 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810186 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810257 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810289 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810325 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\" (UniqueName: \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810364 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\") pod \"09f6563c-411e-4018-b4e0-531f774367f0\" (UID: \"09f6563c-411e-4018-b4e0-531f774367f0\") " Oct 30 20:00:51.810540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810467 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:51.810540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810491 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:51.810791 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.810756 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:51.811054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.811032 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:51.811150 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.811056 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:51.811150 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.811121 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:51.812587 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.812562 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:51.812677 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.812609 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "kube-api-access-2fx5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:51.812720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.812702 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs" pod "09f6563c-411e-4018-b4e0-531f774367f0" (UID: "09f6563c-411e-4018-b4e0-531f774367f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:51.911926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.911844 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f6563c-411e-4018-b4e0-531f774367f0-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:51.911926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.911874 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f6563c-411e-4018-b4e0-531f774367f0-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:51.911926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:51.911887 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fx5h\" (UniqueName: \"kubernetes.io/projected/09f6563c-411e-4018-b4e0-531f774367f0-kube-api-access-2fx5h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:52.028160 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.028123 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp"] Oct 30 20:00:52.032276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.032225 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effe227d-s9zpp"] Oct 30 20:00:52.223109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.223007 4990 generic.go:356] "Generic (PLEG): container finished" podID="09f6563c-411e-4018-b4e0-531f774367f0" containerID="02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e" exitCode=0 Oct 30 20:00:52.223109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.223055 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" Oct 30 20:00:52.223109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.223062 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" event={"ID":"09f6563c-411e-4018-b4e0-531f774367f0","Type":"ContainerDied","Data":"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e"} Oct 30 20:00:52.223109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.223109 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" event={"ID":"09f6563c-411e-4018-b4e0-531f774367f0","Type":"ContainerDied","Data":"d65ccddc32aba252a78616b63190b936ac5c95860f482358a352c0a5220419b6"} Oct 30 20:00:52.223519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.223128 4990 scope.go:122] "RemoveContainer" containerID="02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e" Oct 30 20:00:52.237643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.237623 4990 scope.go:122] "RemoveContainer" containerID="02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e" Oct 30 20:00:52.237919 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:52.237897 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e\": container with ID starting with 02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e not found: ID does not exist" containerID="02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e" Oct 30 20:00:52.237972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.237927 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e"} err="failed to get container status \"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e\": rpc error: code = NotFound desc = could not find container \"02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e\": container with ID starting with 02bed277a761c6c57bcf00a81a7b9b2d322718bcf25c94e6dcb9e5b16363925e not found: ID does not exist" Oct 30 20:00:52.270272 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.270229 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 20:00:52.274510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.274488 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd"] Oct 30 20:00:52.402705 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.402675 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f6563c-411e-4018-b4e0-531f774367f0" path="/var/lib/kubelet/pods/09f6563c-411e-4018-b4e0-531f774367f0/volumes" Oct 30 20:00:52.403177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.403162 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a8eda6-66a4-47db-a4b2-c9f2c6e424b7" path="/var/lib/kubelet/pods/42a8eda6-66a4-47db-a4b2-c9f2c6e424b7/volumes" Oct 30 20:00:52.753323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.753274 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-ed2ede/ipp-route-ed2ede-kserve-769456bcb5-pgngd" podUID="09f6563c-411e-4018-b4e0-531f774367f0" containerName="main" probeResult="failure" output="Get \"https://10.134.1.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:52.822891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:52.819896 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.258990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.258899 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.434224 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.432895 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.472119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.471953 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 20:00:54.505854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.505744 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 20:00:54.560495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.560461 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\" (UniqueName: \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\") pod \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " Oct 30 20:00:54.561228 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.561202 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\" (UniqueName: \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\") pod \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " Oct 30 20:00:54.561757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.561740 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\") pod \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\" (UID: \"8ae924eb-1d67-4a66-86d5-a09626b1aeaa\") " Oct 30 20:00:54.562331 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.562310 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\") pod \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\" (UID: \"54b5317d-dff4-4cd2-8f08-ce5b5066cac3\") " Oct 30 20:00:54.563593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.562235 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume" pod "8ae924eb-1d67-4a66-86d5-a09626b1aeaa" (UID: "8ae924eb-1d67-4a66-86d5-a09626b1aeaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:54.564280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.564256 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume" pod "54b5317d-dff4-4cd2-8f08-ce5b5066cac3" (UID: "54b5317d-dff4-4cd2-8f08-ce5b5066cac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:54.565685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.564548 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq" pod "54b5317d-dff4-4cd2-8f08-ce5b5066cac3" (UID: "54b5317d-dff4-4cd2-8f08-ce5b5066cac3"). InnerVolumeSpecName "kube-api-access-bzqkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:54.571202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.571168 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j" pod "8ae924eb-1d67-4a66-86d5-a09626b1aeaa" (UID: "8ae924eb-1d67-4a66-86d5-a09626b1aeaa"). InnerVolumeSpecName "kube-api-access-4lh9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:54.579812 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.579744 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.668646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.666849 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzqkq\" (UniqueName: \"kubernetes.io/projected/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-kube-api-access-bzqkq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:54.668646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.666888 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh9j\" (UniqueName: \"kubernetes.io/projected/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-kube-api-access-4lh9j\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:54.668646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.666906 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae924eb-1d67-4a66-86d5-a09626b1aeaa-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:54.668646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.666922 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b5317d-dff4-4cd2-8f08-ce5b5066cac3-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:00:54.737165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.735524 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.755456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:54.755412 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" containerID="cri-o://42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f" gracePeriod=25 Oct 30 20:00:55.252501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.252466 4990 generic.go:356] "Generic (PLEG): container finished" podID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerID="d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949" exitCode=0 Oct 30 20:00:55.252715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.252521 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" Oct 30 20:00:55.252715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.252559 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" event={"ID":"8ae924eb-1d67-4a66-86d5-a09626b1aeaa","Type":"ContainerDied","Data":"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949"} Oct 30 20:00:55.252715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.252620 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4" event={"ID":"8ae924eb-1d67-4a66-86d5-a09626b1aeaa","Type":"ContainerDied","Data":"e379693dc7e41b7ddf5c649afb5a935bd8198aa4165eb4c3de35e277d291ac7f"} Oct 30 20:00:55.252715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.252649 4990 scope.go:122] "RemoveContainer" containerID="d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949" Oct 30 20:00:55.254329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.254306 4990 generic.go:356] "Generic (PLEG): container finished" podID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerID="42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740" exitCode=0 Oct 30 20:00:55.254452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.254351 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" Oct 30 20:00:55.254452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.254384 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" event={"ID":"54b5317d-dff4-4cd2-8f08-ce5b5066cac3","Type":"ContainerDied","Data":"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740"} Oct 30 20:00:55.254452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.254429 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td" event={"ID":"54b5317d-dff4-4cd2-8f08-ce5b5066cac3","Type":"ContainerDied","Data":"a13ec9bf4e2f4652a154b2ac2132a9676664718a94f3847795211f27dba60b7b"} Oct 30 20:00:55.265014 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.264994 4990 scope.go:122] "RemoveContainer" containerID="d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949" Oct 30 20:00:55.265369 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:55.265342 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949\": container with ID starting with d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949 not found: ID does not exist" containerID="d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949" Oct 30 20:00:55.265473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.265377 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949"} err="failed to get container status \"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949\": rpc error: code = NotFound desc = could not find container \"d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949\": container with ID starting with d213e120af2e466234bbdcc6d1b3023160b1907b6443ed6b22ab9903cafdd949 not found: ID does not exist" Oct 30 20:00:55.265473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.265401 4990 scope.go:122] "RemoveContainer" containerID="42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740" Oct 30 20:00:55.275628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.275607 4990 scope.go:122] "RemoveContainer" containerID="42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740" Oct 30 20:00:55.275937 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:00:55.275911 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740\": container with ID starting with 42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740 not found: ID does not exist" containerID="42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740" Oct 30 20:00:55.276042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.275941 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740"} err="failed to get container status \"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740\": rpc error: code = NotFound desc = could not find container \"42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740\": container with ID starting with 42f116f21727658a39b560b955b6d9c0e1059c95f9745541999de84f19db9740 not found: ID does not exist" Oct 30 20:00:55.298539 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.298513 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 20:00:55.310286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.310258 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-f83282-78b5d7f44b-8d5td"] Oct 30 20:00:55.333433 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.333402 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 20:00:55.336934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.336906 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-f83282-5f864465ff-skvn4"] Oct 30 20:00:55.603389 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:55.603355 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:56.082817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:56.082148 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.404087 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:56.404009 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" path="/var/lib/kubelet/pods/54b5317d-dff4-4cd2-8f08-ce5b5066cac3/volumes" Oct 30 20:00:56.404501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:56.404484 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" path="/var/lib/kubelet/pods/8ae924eb-1d67-4a66-86d5-a09626b1aeaa/volumes" Oct 30 20:00:57.147901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.147871 4990 scope.go:122] "RemoveContainer" containerID="ca22388381d8a88feeb81fbcd135a8c4379cde487faaace02b5f68e9d9f00c77" Oct 30 20:00:57.159103 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.159066 4990 scope.go:122] "RemoveContainer" containerID="b45215b2e07c7af77c1a3fcd6fe1374912570c79a89bc99971561e230f39b14e" Oct 30 20:00:57.170876 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.170858 4990 scope.go:122] "RemoveContainer" containerID="f7510a936621af5a534a2aacda97585cfdec0a27b8316b2a1d9c7789f3d5d63f" Oct 30 20:00:57.178857 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.178835 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:00:57.178989 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.178913 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:00:57.697239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:00:57.695916 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:00.135766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:00.135714 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.888830 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.888792 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2"] Oct 30 20:01:01.889800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889778 4990 cpu_manager.go:414] "containerMap: removing container" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" Oct 30 20:01:01.889851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889803 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" Oct 30 20:01:01.889851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889833 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" containerName="maas-api" Oct 30 20:01:01.889851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889841 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" containerName="maas-api" Oct 30 20:01:01.889970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889887 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" Oct 30 20:01:01.889970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889895 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" Oct 30 20:01:01.889970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889907 4990 cpu_manager.go:414] "containerMap: removing container" podUID="09f6563c-411e-4018-b4e0-531f774367f0" containerName="main" Oct 30 20:01:01.889970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.889914 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f6563c-411e-4018-b4e0-531f774367f0" containerName="main" Oct 30 20:01:01.890140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.890058 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b5317d-dff4-4cd2-8f08-ce5b5066cac3" containerName="payload-processing" Oct 30 20:01:01.890140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.890075 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d456f4c8-a2f1-4ad3-8f6e-19ac98065606" containerName="maas-api" Oct 30 20:01:01.890140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.890105 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae924eb-1d67-4a66-86d5-a09626b1aeaa" containerName="payload-pre-processing" Oct 30 20:01:01.890140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.890113 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f6563c-411e-4018-b4e0-531f774367f0" containerName="main" Oct 30 20:01:01.898858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.898835 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:01.910767 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.910742 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw"] Oct 30 20:01:01.915583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.915555 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:01.917306 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.917283 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2"] Oct 30 20:01:01.917423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.917408 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:01.918191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.918166 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:01:01.941709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.941665 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw"] Oct 30 20:01:01.968718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.968568 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:01.968718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.968669 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:01.968986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.968786 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:01.968986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.968887 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:01.969137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.968999 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77m9\" (UniqueName: \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:01.969260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:01.969236 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcbl\" (UniqueName: \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.070630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.070588 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.070845 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.070665 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.070845 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.070715 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.070845 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.070792 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77m9\" (UniqueName: \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.071023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.070962 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcbl\" (UniqueName: \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.071023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.071001 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.071484 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.071448 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.071633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.071483 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.073322 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.073295 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.073637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.073616 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.087775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.087744 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77m9\" (UniqueName: \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\") pod \"maas-api-revoke-sub-06a6026c-9lgh2\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.088251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.088229 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcbl\" (UniqueName: \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\") pod \"maas-api-revoke-sub-ed9457c9-j8gfw\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.220201 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.220080 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:02.232427 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.232394 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:02.259563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.255393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.556807 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.556776 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2"] Oct 30 20:01:02.558615 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:02.558588 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c10a45_480b_45d6_ab0b_7c412a77816b.slice/crio-326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a WatchSource:0}: Error finding container 326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a: Status 404 returned error can't find the container with id 326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a Oct 30 20:01:02.594621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:02.594599 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw"] Oct 30 20:01:02.598643 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:02.598618 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca5b25c_a234_4c92_8685_ab02abd021f4.slice/crio-dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8 WatchSource:0}: Error finding container dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8: Status 404 returned error can't find the container with id dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8 Oct 30 20:01:03.066128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.065464 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.215236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.215195 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.353599 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.353560 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.360756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.360722 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" event={"ID":"eca5b25c-a234-4c92-8685-ab02abd021f4","Type":"ContainerStarted","Data":"c05978f903f9a54e8b8c7787aaa640d644a11f5c6ddccdeabdbf5fedc67eca03"} Oct 30 20:01:03.360917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.360788 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" event={"ID":"eca5b25c-a234-4c92-8685-ab02abd021f4","Type":"ContainerStarted","Data":"dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8"} Oct 30 20:01:03.362945 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.362913 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" event={"ID":"a2c10a45-480b-45d6-ab0b-7c412a77816b","Type":"ContainerStarted","Data":"3b202356aaec9518363c87eb7e135aaffdf45e3958abccfe7988f2ae5240be37"} Oct 30 20:01:03.363072 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.362967 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" event={"ID":"a2c10a45-480b-45d6-ab0b-7c412a77816b","Type":"ContainerStarted","Data":"326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a"} Oct 30 20:01:03.396903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.396829 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" podStartSLOduration=2.396810139 podStartE2EDuration="2.396810139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:03.394519297 +0000 UTC m=+3371.106044663" watchObservedRunningTime="2026-10-30 20:01:03.396810139 +0000 UTC m=+3371.108335505" Oct 30 20:01:03.430640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.430584 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" podStartSLOduration=2.43056868 podStartE2EDuration="2.43056868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:03.427391279 +0000 UTC m=+3371.138916646" watchObservedRunningTime="2026-10-30 20:01:03.43056868 +0000 UTC m=+3371.142094045" Oct 30 20:01:03.891856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:03.891772 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.373248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:04.373213 4990 generic.go:356] "Generic (PLEG): container finished" podID="a2c10a45-480b-45d6-ab0b-7c412a77816b" containerID="3b202356aaec9518363c87eb7e135aaffdf45e3958abccfe7988f2ae5240be37" exitCode=0 Oct 30 20:01:04.373729 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:04.373302 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" event={"ID":"a2c10a45-480b-45d6-ab0b-7c412a77816b","Type":"ContainerDied","Data":"3b202356aaec9518363c87eb7e135aaffdf45e3958abccfe7988f2ae5240be37"} Oct 30 20:01:04.375185 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:04.375166 4990 generic.go:356] "Generic (PLEG): container finished" podID="eca5b25c-a234-4c92-8685-ab02abd021f4" containerID="c05978f903f9a54e8b8c7787aaa640d644a11f5c6ddccdeabdbf5fedc67eca03" exitCode=0 Oct 30 20:01:04.375276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:04.375217 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" event={"ID":"eca5b25c-a234-4c92-8685-ab02abd021f4","Type":"ContainerDied","Data":"c05978f903f9a54e8b8c7787aaa640d644a11f5c6ddccdeabdbf5fedc67eca03"} Oct 30 20:01:05.121280 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:05.121242 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:01:05.439037 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.438959 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.624111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.623481 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:01:05.734443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.733227 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.740274 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.740233 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:05.743567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.742585 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:05.832886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.832854 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\" (UniqueName: \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\") pod \"eca5b25c-a234-4c92-8685-ab02abd021f4\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " Oct 30 20:01:05.833263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.833243 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\") pod \"eca5b25c-a234-4c92-8685-ab02abd021f4\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " Oct 30 20:01:05.833396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.833382 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\") pod \"a2c10a45-480b-45d6-ab0b-7c412a77816b\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " Oct 30 20:01:05.833529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.833516 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\" (UniqueName: \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\") pod \"a2c10a45-480b-45d6-ab0b-7c412a77816b\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " Oct 30 20:01:05.833732 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.833719 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\") pod \"a2c10a45-480b-45d6-ab0b-7c412a77816b\" (UID: \"a2c10a45-480b-45d6-ab0b-7c412a77816b\") " Oct 30 20:01:05.833841 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.833829 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\") pod \"eca5b25c-a234-4c92-8685-ab02abd021f4\" (UID: \"eca5b25c-a234-4c92-8685-ab02abd021f4\") " Oct 30 20:01:05.835383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.835126 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca" pod "a2c10a45-480b-45d6-ab0b-7c412a77816b" (UID: "a2c10a45-480b-45d6-ab0b-7c412a77816b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:05.850142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.835737 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca" pod "eca5b25c-a234-4c92-8685-ab02abd021f4" (UID: "eca5b25c-a234-4c92-8685-ab02abd021f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:05.850142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.839078 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9" pod "a2c10a45-480b-45d6-ab0b-7c412a77816b" (UID: "a2c10a45-480b-45d6-ab0b-7c412a77816b"). InnerVolumeSpecName "kube-api-access-s77m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:05.850552 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.850486 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl" pod "eca5b25c-a234-4c92-8685-ab02abd021f4" (UID: "eca5b25c-a234-4c92-8685-ab02abd021f4"). InnerVolumeSpecName "kube-api-access-2fcbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:05.853832 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.853797 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca" pod "eca5b25c-a234-4c92-8685-ab02abd021f4" (UID: "eca5b25c-a234-4c92-8685-ab02abd021f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:05.856133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.855844 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca" pod "a2c10a45-480b-45d6-ab0b-7c412a77816b" (UID: "a2c10a45-480b-45d6-ab0b-7c412a77816b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:05.935432 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935398 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fcbl\" (UniqueName: \"kubernetes.io/projected/eca5b25c-a234-4c92-8685-ab02abd021f4-kube-api-access-2fcbl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:05.935432 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935427 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:05.935432 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935440 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:05.935678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935449 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77m9\" (UniqueName: \"kubernetes.io/projected/a2c10a45-480b-45d6-ab0b-7c412a77816b-kube-api-access-s77m9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:05.935678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935458 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2c10a45-480b-45d6-ab0b-7c412a77816b-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:05.935678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:05.935467 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5b25c-a234-4c92-8685-ab02abd021f4-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:06.216605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.216565 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.420798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.420763 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" Oct 30 20:01:06.420798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.420787 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw" event={"ID":"eca5b25c-a234-4c92-8685-ab02abd021f4","Type":"ContainerDied","Data":"dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8"} Oct 30 20:01:06.421027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.420830 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd88e357a559764a25e80a99de205c5cf7dcc033b45aba14ea25f60d93b13fc8" Oct 30 20:01:06.423288 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.423265 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" Oct 30 20:01:06.423462 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.423298 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2" event={"ID":"a2c10a45-480b-45d6-ab0b-7c412a77816b","Type":"ContainerDied","Data":"326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a"} Oct 30 20:01:06.423462 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:06.423341 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326e603e1e3d22ac996b5104517d10bf2c6f0a5fed73516f1ad8a81568dd429a" Oct 30 20:01:07.656166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:07.654402 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.179030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:09.178986 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.040343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.040311 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr"] Oct 30 20:01:10.041009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.040989 4990 cpu_manager.go:414] "containerMap: removing container" podUID="eca5b25c-a234-4c92-8685-ab02abd021f4" containerName="revoke-keys" Oct 30 20:01:10.041009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.041007 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca5b25c-a234-4c92-8685-ab02abd021f4" containerName="revoke-keys" Oct 30 20:01:10.041209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.041038 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a2c10a45-480b-45d6-ab0b-7c412a77816b" containerName="revoke-keys" Oct 30 20:01:10.041209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.041044 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c10a45-480b-45d6-ab0b-7c412a77816b" containerName="revoke-keys" Oct 30 20:01:10.041209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.041168 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c10a45-480b-45d6-ab0b-7c412a77816b" containerName="revoke-keys" Oct 30 20:01:10.041209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.041183 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca5b25c-a234-4c92-8685-ab02abd021f4" containerName="revoke-keys" Oct 30 20:01:10.045078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.045051 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.062558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.062532 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr"] Oct 30 20:01:10.078114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.077518 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:01:10.078114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.077587 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:10.115364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.115333 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.115553 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.115487 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.115633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.115571 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvn9w\" (UniqueName: \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.217687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.217638 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.218189 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.217849 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.218189 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.217890 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvn9w\" (UniqueName: \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.218441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.218417 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.220897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.220868 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.235373 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.235341 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvn9w\" (UniqueName: \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.360507 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.360476 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:10.691552 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.691521 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr"] Oct 30 20:01:10.693414 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:10.693386 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241668ac_58e4_4d30_9150_be66377cd66e.slice/crio-4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1 WatchSource:0}: Error finding container 4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1: Status 404 returned error can't find the container with id 4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1 Oct 30 20:01:10.916233 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:10.916192 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:11.497422 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:11.497380 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" event={"ID":"241668ac-58e4-4d30-9150-be66377cd66e","Type":"ContainerStarted","Data":"c7b6c339a305d967e395ec9e7cf05dd43c2e21e0e4c782772aafbe791cca49a7"} Oct 30 20:01:11.497884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:11.497444 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" event={"ID":"241668ac-58e4-4d30-9150-be66377cd66e","Type":"ContainerStarted","Data":"4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1"} Oct 30 20:01:11.535629 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:11.535578 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" podStartSLOduration=1.53556478 podStartE2EDuration="1.53556478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:11.534516253 +0000 UTC m=+3379.246041618" watchObservedRunningTime="2026-10-30 20:01:11.53556478 +0000 UTC m=+3379.247090172" Oct 30 20:01:12.510726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:12.510691 4990 generic.go:356] "Generic (PLEG): container finished" podID="241668ac-58e4-4d30-9150-be66377cd66e" containerID="c7b6c339a305d967e395ec9e7cf05dd43c2e21e0e4c782772aafbe791cca49a7" exitCode=0 Oct 30 20:01:12.511256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:12.510781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" event={"ID":"241668ac-58e4-4d30-9150-be66377cd66e","Type":"ContainerDied","Data":"c7b6c339a305d967e395ec9e7cf05dd43c2e21e0e4c782772aafbe791cca49a7"} Oct 30 20:01:12.772611 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:12.772512 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.919504 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:12.919466 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.695863 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.695841 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:13.775930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.774679 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\" (UniqueName: \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\") pod \"241668ac-58e4-4d30-9150-be66377cd66e\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " Oct 30 20:01:13.775930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.774749 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\") pod \"241668ac-58e4-4d30-9150-be66377cd66e\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " Oct 30 20:01:13.775930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.775071 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\") pod \"241668ac-58e4-4d30-9150-be66377cd66e\" (UID: \"241668ac-58e4-4d30-9150-be66377cd66e\") " Oct 30 20:01:13.775930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.775610 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca" pod "241668ac-58e4-4d30-9150-be66377cd66e" (UID: "241668ac-58e4-4d30-9150-be66377cd66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:13.775930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.775776 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:13.778058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.778024 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w" pod "241668ac-58e4-4d30-9150-be66377cd66e" (UID: "241668ac-58e4-4d30-9150-be66377cd66e"). InnerVolumeSpecName "kube-api-access-nvn9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:13.778503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.778483 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca" pod "241668ac-58e4-4d30-9150-be66377cd66e" (UID: "241668ac-58e4-4d30-9150-be66377cd66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:13.877231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.877201 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvn9w\" (UniqueName: \"kubernetes.io/projected/241668ac-58e4-4d30-9150-be66377cd66e-kube-api-access-nvn9w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:13.877231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:13.877234 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241668ac-58e4-4d30-9150-be66377cd66e-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:14.428204 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.428172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.530537 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.530509 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" Oct 30 20:01:14.530720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.530544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr" event={"ID":"241668ac-58e4-4d30-9150-be66377cd66e","Type":"ContainerDied","Data":"4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1"} Oct 30 20:01:14.530720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.530594 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eff4188ef7e69db913d5d892b9dbfaf33b2c5cc6e27f37e039d52d01c9b3cd1" Oct 30 20:01:14.532467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.532442 4990 generic.go:356] "Generic (PLEG): container finished" podID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerID="1afaf109347cfc56597aa7cb458c9d92a79877f637efb5b5808ddb4c3034d2d4" exitCode=0 Oct 30 20:01:14.532628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.532510 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerDied","Data":"1afaf109347cfc56597aa7cb458c9d92a79877f637efb5b5808ddb4c3034d2d4"} Oct 30 20:01:14.532973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.532950 4990 scope.go:122] "RemoveContainer" containerID="1afaf109347cfc56597aa7cb458c9d92a79877f637efb5b5808ddb4c3034d2d4" Oct 30 20:01:14.534584 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.534557 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:01:14.534664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.534625 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:01:14.534664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.534659 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:01:14.924136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:14.923526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.077606 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.077576 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 20:01:15.196953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.196915 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\" (UniqueName: \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\") pod \"6ac42b72-87de-4b3c-b813-a0297d8f0187\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " Oct 30 20:01:15.197176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.197024 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\") pod \"6ac42b72-87de-4b3c-b813-a0297d8f0187\" (UID: \"6ac42b72-87de-4b3c-b813-a0297d8f0187\") " Oct 30 20:01:15.197415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.197391 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume" pod "6ac42b72-87de-4b3c-b813-a0297d8f0187" (UID: "6ac42b72-87de-4b3c-b813-a0297d8f0187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:15.197618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.197602 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac42b72-87de-4b3c-b813-a0297d8f0187-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:15.199298 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.199278 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn" pod "6ac42b72-87de-4b3c-b813-a0297d8f0187" (UID: "6ac42b72-87de-4b3c-b813-a0297d8f0187"). InnerVolumeSpecName "kube-api-access-lqnrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:15.221407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.221385 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:01:15.298733 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.298672 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnrn\" (UniqueName: \"kubernetes.io/projected/6ac42b72-87de-4b3c-b813-a0297d8f0187-kube-api-access-lqnrn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:15.541438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.541399 4990 generic.go:356] "Generic (PLEG): container finished" podID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerID="42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f" exitCode=0 Oct 30 20:01:15.541633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.541452 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" Oct 30 20:01:15.541633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.541485 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" event={"ID":"6ac42b72-87de-4b3c-b813-a0297d8f0187","Type":"ContainerDied","Data":"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f"} Oct 30 20:01:15.541633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.541541 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f" event={"ID":"6ac42b72-87de-4b3c-b813-a0297d8f0187","Type":"ContainerDied","Data":"7601a895e17fc5b7168ab8da0a7b4ee4cee992f3d26c17d6ae4c037e87936ff6"} Oct 30 20:01:15.541633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.541564 4990 scope.go:122] "RemoveContainer" containerID="42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f" Oct 30 20:01:15.544041 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.544011 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerStarted","Data":"b0fa8a3063ce52f7f5fc1016f59b2ecfa4b18da3deb1a54abbbbe1431eaaa8db"} Oct 30 20:01:15.553639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.553619 4990 scope.go:122] "RemoveContainer" containerID="42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f" Oct 30 20:01:15.553927 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:15.553907 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f\": container with ID starting with 42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f not found: ID does not exist" containerID="42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f" Oct 30 20:01:15.553992 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.553939 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f"} err="failed to get container status \"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f\": rpc error: code = NotFound desc = could not find container \"42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f\": container with ID starting with 42ef81839fd982239c867198b45c7ac5542bc41df794b73d918b41e2dbfacd9f not found: ID does not exist" Oct 30 20:01:15.605449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.605410 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 20:01:15.615519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:15.615482 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac614-ddb778cf5-rd79f"] Oct 30 20:01:16.405354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:16.405316 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" path="/var/lib/kubelet/pods/6ac42b72-87de-4b3c-b813-a0297d8f0187/volumes" Oct 30 20:01:16.481424 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:16.481391 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:01:16.486324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:16.485972 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 20:01:16.486324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:16.486022 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 20:01:16.777407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:16.776650 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.482117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:17.481120 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 20:01:17.482117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:17.481181 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 20:01:18.481488 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:18.481451 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 20:01:18.481797 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:18.481766 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 20:01:18.849443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:18.849387 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.484116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:19.483521 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 20:01:19.484116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:19.483588 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 20:01:20.096106 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.096049 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.298235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.298192 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.437648 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.437557 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 20:01:20.437925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.437875 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" containerID="cri-o://c71b3c642a7917ebb08f2f6bfc934104ffe51aaa93a3c04bd19733a2bda28e48" gracePeriod=30 Oct 30 20:01:20.492930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.492894 4990 patch_prober.go:29] interesting pod/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body= Oct 30 20:01:20.495334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.495272 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" Oct 30 20:01:20.497455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.497427 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.658590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.658532 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.692156 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.692007 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 20:01:20.829675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.829633 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.921358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.921322 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d"] Oct 30 20:01:20.922005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.921988 4990 cpu_manager.go:414] "containerMap: removing container" podUID="241668ac-58e4-4d30-9150-be66377cd66e" containerName="revoke-keys" Oct 30 20:01:20.922071 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.922007 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241668ac-58e4-4d30-9150-be66377cd66e" containerName="revoke-keys" Oct 30 20:01:20.922071 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.922028 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" Oct 30 20:01:20.922071 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.922035 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" Oct 30 20:01:20.922315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.922157 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="241668ac-58e4-4d30-9150-be66377cd66e" containerName="revoke-keys" Oct 30 20:01:20.922315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.922169 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac42b72-87de-4b3c-b813-a0297d8f0187" containerName="payload-pre-processing" Oct 30 20:01:20.927281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.927258 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:20.944874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.944794 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d"] Oct 30 20:01:20.986787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:20.986746 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.001892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.001861 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:21.002052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.001993 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:01:21.013834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.013795 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.013982 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.013904 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.014067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.013999 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf24q\" (UniqueName: \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.115712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.115671 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.116199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.115827 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf24q\" (UniqueName: \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.116199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.115954 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.116602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.116582 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.118080 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.118058 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.132519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.132496 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf24q\" (UniqueName: \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\") pod \"maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.244142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.244029 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:21.347113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.339853 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.481001 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.480965 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 20:01:21.485357 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.485328 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:01:21.486400 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.486375 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:01:21.486918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.486899 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:01:21.518889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.517606 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 20:01:21.578941 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.577717 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d"] Oct 30 20:01:21.583953 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:21.583515 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c9e9db_10cd_423e_9364_115bc36770a6.slice/crio-596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966 WatchSource:0}: Error finding container 596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966: Status 404 returned error can't find the container with id 596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966 Oct 30 20:01:21.653066 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.653029 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" event={"ID":"49c9e9db-10cd-423e-9364-115bc36770a6","Type":"ContainerStarted","Data":"596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966"} Oct 30 20:01:21.666236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.666205 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.813575 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:21.813540 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.119865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.119829 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.546162 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.545440 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.663900 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.663865 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" event={"ID":"49c9e9db-10cd-423e-9364-115bc36770a6","Type":"ContainerStarted","Data":"acc6d2c94ae72906d8ef715a1d61bd4097707a98562c94bac70297b0f403b933"} Oct 30 20:01:22.698403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.698372 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.703735 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.702327 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" podStartSLOduration=2.702309409 podStartE2EDuration="2.702309409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:22.69878957 +0000 UTC m=+3390.410314936" watchObservedRunningTime="2026-10-30 20:01:22.702309409 +0000 UTC m=+3390.413834776" Oct 30 20:01:22.870907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.870870 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.978548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.978462 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr"] Oct 30 20:01:22.992756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:22.992712 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cd4bde-9krxr"] Oct 30 20:01:23.015128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.014795 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.270465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.270386 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj_845c4764-992b-445c-8efd-6198b7243572/payload-processing/0.log" Oct 30 20:01:23.678840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.678798 4990 generic.go:356] "Generic (PLEG): container finished" podID="49c9e9db-10cd-423e-9364-115bc36770a6" containerID="acc6d2c94ae72906d8ef715a1d61bd4097707a98562c94bac70297b0f403b933" exitCode=0 Oct 30 20:01:23.679037 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.678843 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" event={"ID":"49c9e9db-10cd-423e-9364-115bc36770a6","Type":"ContainerDied","Data":"acc6d2c94ae72906d8ef715a1d61bd4097707a98562c94bac70297b0f403b933"} Oct 30 20:01:23.928389 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.928348 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2"] Oct 30 20:01:23.932919 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.932856 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:23.950003 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.949978 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2"] Oct 30 20:01:23.960196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.960167 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:23.960349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.960324 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9pk\" (UniqueName: \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:23.960491 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:23.960475 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.061877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.061839 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.062065 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.062032 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.062194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.062174 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9pk\" (UniqueName: \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.062543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.062512 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.064470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.064445 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.078955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.078926 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9pk\" (UniqueName: \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\") pod \"maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.249315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.249238 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:24.404968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.404934 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241668ac-58e4-4d30-9150-be66377cd66e" path="/var/lib/kubelet/pods/241668ac-58e4-4d30-9150-be66377cd66e/volumes" Oct 30 20:01:24.555931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.555906 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2"] Oct 30 20:01:24.558078 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:24.558050 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8049c7ef_e6a3_4095_82c1_dee826dbcd59.slice/crio-3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d WatchSource:0}: Error finding container 3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d: Status 404 returned error can't find the container with id 3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d Oct 30 20:01:24.689941 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.689901 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" event={"ID":"8049c7ef-e6a3-4095-82c1-dee826dbcd59","Type":"ContainerStarted","Data":"3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d"} Oct 30 20:01:24.827610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.827586 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:24.872954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.872918 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\") pod \"49c9e9db-10cd-423e-9364-115bc36770a6\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " Oct 30 20:01:24.873227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.873185 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\" (UniqueName: \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\") pod \"49c9e9db-10cd-423e-9364-115bc36770a6\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " Oct 30 20:01:24.873344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.873326 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\") pod \"49c9e9db-10cd-423e-9364-115bc36770a6\" (UID: \"49c9e9db-10cd-423e-9364-115bc36770a6\") " Oct 30 20:01:24.873414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.873347 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca" pod "49c9e9db-10cd-423e-9364-115bc36770a6" (UID: "49c9e9db-10cd-423e-9364-115bc36770a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:24.873805 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.873784 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:24.875924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.875894 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q" pod "49c9e9db-10cd-423e-9364-115bc36770a6" (UID: "49c9e9db-10cd-423e-9364-115bc36770a6"). InnerVolumeSpecName "kube-api-access-tf24q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:24.876057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.876026 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca" pod "49c9e9db-10cd-423e-9364-115bc36770a6" (UID: "49c9e9db-10cd-423e-9364-115bc36770a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:24.975979 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.975942 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf24q\" (UniqueName: \"kubernetes.io/projected/49c9e9db-10cd-423e-9364-115bc36770a6-kube-api-access-tf24q\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:24.975979 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:24.975978 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c9e9db-10cd-423e-9364-115bc36770a6-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:25.030889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.030851 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.694020 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.693981 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" podUID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" containerName="payload-pre-processing" containerID="cri-o://fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9" gracePeriod=25 Oct 30 20:01:25.707751 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.707694 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" Oct 30 20:01:25.707751 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.707706 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d" event={"ID":"49c9e9db-10cd-423e-9364-115bc36770a6","Type":"ContainerDied","Data":"596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966"} Oct 30 20:01:25.707934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.707768 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596f0c0ed9bf09c57a2d7db99ea105ab28ead03409776363a2296dd9bd883966" Oct 30 20:01:25.710826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.710659 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" event={"ID":"8049c7ef-e6a3-4095-82c1-dee826dbcd59","Type":"ContainerStarted","Data":"04488aa46046fc6e1be67f73ffab9a9509fc01bae3e573a3fb49b81cb05acc3a"} Oct 30 20:01:25.713537 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.713511 4990 generic.go:356] "Generic (PLEG): container finished" podID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerID="c71b3c642a7917ebb08f2f6bfc934104ffe51aaa93a3c04bd19733a2bda28e48" exitCode=0 Oct 30 20:01:25.713653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.713582 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerDied","Data":"c71b3c642a7917ebb08f2f6bfc934104ffe51aaa93a3c04bd19733a2bda28e48"} Oct 30 20:01:25.713653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.713632 4990 scope.go:122] "RemoveContainer" containerID="12755a23009f83152b1610371f670824badf48a24b7ea796e9eb442c67a73ea8" Oct 30 20:01:25.838828 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.838801 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903505 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903585 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903665 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903808 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903840 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903870 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903909 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.903964 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904004 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904069 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\") pod \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\" (UID: \"d691bb5f-4b13-4d10-9349-c4666c2c0f26\") " Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904197 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904474 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904732 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904752 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.904765 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.905063 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:25.906176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.905629 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:25.909810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.909771 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:25.911539 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.910888 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:25.911539 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.911311 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "kube-api-access-hm2xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:25.914421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.912209 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token" pod "d691bb5f-4b13-4d10-9349-c4666c2c0f26" (UID: "d691bb5f-4b13-4d10-9349-c4666c2c0f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:25.969974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:25.969947 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 20:01:26.005422 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.005394 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\" (UniqueName: \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\") pod \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " Oct 30 20:01:26.005539 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.005526 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\") pod \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\" (UID: \"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b\") " Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.006328 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume" pod "7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" (UID: "7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017728 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017760 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2xl\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-kube-api-access-hm2xl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017775 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017795 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017807 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017820 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d691bb5f-4b13-4d10-9349-c4666c2c0f26-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.023759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.017834 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d691bb5f-4b13-4d10-9349-c4666c2c0f26-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.024878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.024833 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg" pod "7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" (UID: "7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b"). InnerVolumeSpecName "kube-api-access-6tfvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:26.119002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.118969 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfvg\" (UniqueName: \"kubernetes.io/projected/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b-kube-api-access-6tfvg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:26.481612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.481565 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" podUID="8e8bf6bf-b063-416d-923b-bcf21e98f692" containerName="payload-processing" containerID="cri-o://534ce59bcf065bda5ac524b47030a50542c88bce4c1cea67fbeffd86f798276b" gracePeriod=25 Oct 30 20:01:26.518180 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.518147 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" podUID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" containerName="payload-pre-processing" containerID="cri-o://5026e2e426fcb9c088cacc659687d28ecd3343651ab653e5744b6b5c159ca61d" gracePeriod=25 Oct 30 20:01:26.565352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.565221 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.731062 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.729787 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" event={"ID":"d691bb5f-4b13-4d10-9349-c4666c2c0f26","Type":"ContainerDied","Data":"e620e6c3ca2cae0d8a79464cb83b139017f6420af6e60404d12eee2d85cc35fe"} Oct 30 20:01:26.731062 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.729841 4990 scope.go:122] "RemoveContainer" containerID="c71b3c642a7917ebb08f2f6bfc934104ffe51aaa93a3c04bd19733a2bda28e48" Oct 30 20:01:26.731062 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.729961 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf" Oct 30 20:01:26.734396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.733712 4990 generic.go:356] "Generic (PLEG): container finished" podID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" containerID="5026e2e426fcb9c088cacc659687d28ecd3343651ab653e5744b6b5c159ca61d" exitCode=0 Oct 30 20:01:26.734396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.733901 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" event={"ID":"4b82cbfb-4630-44b0-83a9-a38a5d48ff73","Type":"ContainerDied","Data":"5026e2e426fcb9c088cacc659687d28ecd3343651ab653e5744b6b5c159ca61d"} Oct 30 20:01:26.736736 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.736711 4990 generic.go:356] "Generic (PLEG): container finished" podID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" containerID="fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9" exitCode=0 Oct 30 20:01:26.736844 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.736812 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" event={"ID":"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b","Type":"ContainerDied","Data":"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9"} Oct 30 20:01:26.736844 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.736841 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" event={"ID":"7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b","Type":"ContainerDied","Data":"13024b5c40bdf14c2846b6638de85adc20202569dcbd999d335d6055d964a53e"} Oct 30 20:01:26.736932 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.736919 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96" Oct 30 20:01:26.741315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.740034 4990 generic.go:356] "Generic (PLEG): container finished" podID="8e8bf6bf-b063-416d-923b-bcf21e98f692" containerID="534ce59bcf065bda5ac524b47030a50542c88bce4c1cea67fbeffd86f798276b" exitCode=0 Oct 30 20:01:26.741315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.740247 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" event={"ID":"8e8bf6bf-b063-416d-923b-bcf21e98f692","Type":"ContainerDied","Data":"534ce59bcf065bda5ac524b47030a50542c88bce4c1cea67fbeffd86f798276b"} Oct 30 20:01:26.742803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.742741 4990 generic.go:356] "Generic (PLEG): container finished" podID="8049c7ef-e6a3-4095-82c1-dee826dbcd59" containerID="04488aa46046fc6e1be67f73ffab9a9509fc01bae3e573a3fb49b81cb05acc3a" exitCode=0 Oct 30 20:01:26.742803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.742778 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" event={"ID":"8049c7ef-e6a3-4095-82c1-dee826dbcd59","Type":"ContainerDied","Data":"04488aa46046fc6e1be67f73ffab9a9509fc01bae3e573a3fb49b81cb05acc3a"} Oct 30 20:01:26.760571 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.760545 4990 scope.go:122] "RemoveContainer" containerID="fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9" Oct 30 20:01:26.773326 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.773080 4990 scope.go:122] "RemoveContainer" containerID="fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9" Oct 30 20:01:26.773657 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:26.773563 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9\": container with ID starting with fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9 not found: ID does not exist" containerID="fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9" Oct 30 20:01:26.773657 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.773603 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9"} err="failed to get container status \"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9\": rpc error: code = NotFound desc = could not find container \"fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9\": container with ID starting with fc84b1c8d7f8be1ad7d0278cb945d45a691ef4738e2c3797e8330624c371b9d9 not found: ID does not exist" Oct 30 20:01:26.823772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.823560 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 20:01:26.829759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.829678 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed2ede-openshift-default-65cdc4f67c-rmmlf"] Oct 30 20:01:26.854058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.853575 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 20:01:26.857556 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.857532 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cd4bde-fff5dbf5f-89z96"] Oct 30 20:01:26.860361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.860341 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 20:01:26.932009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.931973 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\") pod \"8e8bf6bf-b063-416d-923b-bcf21e98f692\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " Oct 30 20:01:26.932197 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.932063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\" (UniqueName: \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\") pod \"8e8bf6bf-b063-416d-923b-bcf21e98f692\" (UID: \"8e8bf6bf-b063-416d-923b-bcf21e98f692\") " Oct 30 20:01:26.932645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.932621 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume" pod "8e8bf6bf-b063-416d-923b-bcf21e98f692" (UID: "8e8bf6bf-b063-416d-923b-bcf21e98f692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:26.935816 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.935792 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n" pod "8e8bf6bf-b063-416d-923b-bcf21e98f692" (UID: "8e8bf6bf-b063-416d-923b-bcf21e98f692"). InnerVolumeSpecName "kube-api-access-8k27n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:26.937766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.937749 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 20:01:26.962907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:26.962886 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:27.033304 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033217 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\") pod \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " Oct 30 20:01:27.033482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033323 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\" (UniqueName: \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\") pod \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " Oct 30 20:01:27.033482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033359 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\") pod \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\" (UID: \"8049c7ef-e6a3-4095-82c1-dee826dbcd59\") " Oct 30 20:01:27.033482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033447 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\" (UniqueName: \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\") pod \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " Oct 30 20:01:27.033675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033535 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\") pod \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\" (UID: \"4b82cbfb-4630-44b0-83a9-a38a5d48ff73\") " Oct 30 20:01:27.033675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.033622 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca" pod "8049c7ef-e6a3-4095-82c1-dee826dbcd59" (UID: "8049c7ef-e6a3-4095-82c1-dee826dbcd59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:27.034233 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.034194 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k27n\" (UniqueName: \"kubernetes.io/projected/8e8bf6bf-b063-416d-923b-bcf21e98f692-kube-api-access-8k27n\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.034233 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.034221 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.034414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.034237 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8bf6bf-b063-416d-923b-bcf21e98f692-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.034475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.034462 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume" pod "4b82cbfb-4630-44b0-83a9-a38a5d48ff73" (UID: "4b82cbfb-4630-44b0-83a9-a38a5d48ff73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:27.036603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.036567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca" pod "8049c7ef-e6a3-4095-82c1-dee826dbcd59" (UID: "8049c7ef-e6a3-4095-82c1-dee826dbcd59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:27.036719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.036628 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x" pod "4b82cbfb-4630-44b0-83a9-a38a5d48ff73" (UID: "4b82cbfb-4630-44b0-83a9-a38a5d48ff73"). InnerVolumeSpecName "kube-api-access-lf72x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:27.036785 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.036705 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk" pod "8049c7ef-e6a3-4095-82c1-dee826dbcd59" (UID: "8049c7ef-e6a3-4095-82c1-dee826dbcd59"). InnerVolumeSpecName "kube-api-access-rh9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:27.135757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.135717 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9pk\" (UniqueName: \"kubernetes.io/projected/8049c7ef-e6a3-4095-82c1-dee826dbcd59-kube-api-access-rh9pk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.135757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.135756 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8049c7ef-e6a3-4095-82c1-dee826dbcd59-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.135757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.135767 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf72x\" (UniqueName: \"kubernetes.io/projected/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-kube-api-access-lf72x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.135986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.135776 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b82cbfb-4630-44b0-83a9-a38a5d48ff73-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:27.751739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.751704 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" Oct 30 20:01:27.752257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.751741 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv" event={"ID":"8e8bf6bf-b063-416d-923b-bcf21e98f692","Type":"ContainerDied","Data":"7864f3328e959310b687e60e1059a59082d6e32c6e41796696b37ca372c2cc96"} Oct 30 20:01:27.752257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.751812 4990 scope.go:122] "RemoveContainer" containerID="534ce59bcf065bda5ac524b47030a50542c88bce4c1cea67fbeffd86f798276b" Oct 30 20:01:27.753468 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.753452 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" Oct 30 20:01:27.753605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.753467 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2" event={"ID":"8049c7ef-e6a3-4095-82c1-dee826dbcd59","Type":"ContainerDied","Data":"3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d"} Oct 30 20:01:27.753605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.753511 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba495b4fabb2b1e1de42c83c9b0c89de568891327b8ef41c98c5b7aac239a7d" Oct 30 20:01:27.756546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.756526 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" Oct 30 20:01:27.756703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.756575 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s" event={"ID":"4b82cbfb-4630-44b0-83a9-a38a5d48ff73","Type":"ContainerDied","Data":"c0653799fe08da59449e233ba2b1130ce8421e086c41dfddbfded57b8bf8ca97"} Oct 30 20:01:27.769414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.766276 4990 scope.go:122] "RemoveContainer" containerID="5026e2e426fcb9c088cacc659687d28ecd3343651ab653e5744b6b5c159ca61d" Oct 30 20:01:27.798535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.798511 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 20:01:27.807871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.807849 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-79cd976c6b-ndzrv"] Oct 30 20:01:27.854423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.854388 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 20:01:27.862441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:27.862412 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-7df9999cf5-9gm7s"] Oct 30 20:01:28.167509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.167473 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.404129 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.404067 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" path="/var/lib/kubelet/pods/4b82cbfb-4630-44b0-83a9-a38a5d48ff73/volumes" Oct 30 20:01:28.404753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.404729 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" path="/var/lib/kubelet/pods/7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b/volumes" Oct 30 20:01:28.405300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.405278 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8bf6bf-b063-416d-923b-bcf21e98f692" path="/var/lib/kubelet/pods/8e8bf6bf-b063-416d-923b-bcf21e98f692/volumes" Oct 30 20:01:28.406360 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.406336 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" path="/var/lib/kubelet/pods/d691bb5f-4b13-4d10-9349-c4666c2c0f26/volumes" Oct 30 20:01:28.836791 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:28.836756 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.943023 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:29.939065 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:32.282017 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:32.281980 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.502892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:33.502852 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.414675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.414628 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.665330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.665231 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.816633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.816591 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.837907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.837873 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4"] Oct 30 20:01:34.845215 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.845190 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac614-j5zk4"] Oct 30 20:01:34.962129 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:34.962021 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.097455 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:35.097413 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:01:35.360921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:35.360885 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:36.402173 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:36.402140 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edc69a9-dfb0-4664-ab2a-fc16f31cf08d" path="/var/lib/kubelet/pods/6edc69a9-dfb0-4664-ab2a-fc16f31cf08d/volumes" Oct 30 20:01:36.668568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:36.668471 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:38.117906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:38.117841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:38.273486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:38.273449 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.231767 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:39.231726 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.407016 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:39.406970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.695376 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:39.695174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.855145 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:39.855102 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.040359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:40.040263 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.270469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:40.270394 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.067397 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:41.067350 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.937377 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:41.937343 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.077869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:42.077826 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.208974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:42.208861 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 20:01:42.209638 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:42.209604 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" containerID="cri-o://79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657" gracePeriod=30 Oct 30 20:01:42.292010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:42.291949 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.722118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:42.722034 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.330893 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:44.330853 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.869452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:45.869387 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.982531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:45.982494 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 20:01:45.982818 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:45.982792 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" podUID="7c170c74-2736-4d97-8b7d-c651fa404435" containerName="payload-pre-processing" containerID="cri-o://7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b" gracePeriod=30 Oct 30 20:01:46.079162 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.079126 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.234884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.234842 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.400266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.400233 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 20:01:46.482987 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.482879 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\" (UniqueName: \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\") pod \"7c170c74-2736-4d97-8b7d-c651fa404435\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " Oct 30 20:01:46.483210 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.483007 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\") pod \"7c170c74-2736-4d97-8b7d-c651fa404435\" (UID: \"7c170c74-2736-4d97-8b7d-c651fa404435\") " Oct 30 20:01:46.483467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.483439 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume" pod "7c170c74-2736-4d97-8b7d-c651fa404435" (UID: "7c170c74-2736-4d97-8b7d-c651fa404435"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:46.483850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.483824 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c170c74-2736-4d97-8b7d-c651fa404435-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:46.485794 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.485744 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv" pod "7c170c74-2736-4d97-8b7d-c651fa404435" (UID: "7c170c74-2736-4d97-8b7d-c651fa404435"). InnerVolumeSpecName "kube-api-access-rwfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:46.585296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.585257 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfdv\" (UniqueName: \"kubernetes.io/projected/7c170c74-2736-4d97-8b7d-c651fa404435-kube-api-access-rwfdv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:46.585870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.585847 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.735619 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.735510 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d"] Oct 30 20:01:46.739293 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.739257 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2afc11-bb88d"] Oct 30 20:01:46.776852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.776806 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.949624 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.949586 4990 generic.go:356] "Generic (PLEG): container finished" podID="7c170c74-2736-4d97-8b7d-c651fa404435" containerID="7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b" exitCode=0 Oct 30 20:01:46.950079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.949639 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" Oct 30 20:01:46.950079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.949672 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" event={"ID":"7c170c74-2736-4d97-8b7d-c651fa404435","Type":"ContainerDied","Data":"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b"} Oct 30 20:01:46.950079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.949736 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc" event={"ID":"7c170c74-2736-4d97-8b7d-c651fa404435","Type":"ContainerDied","Data":"5417e206763d29fdb3d866738fb72751ab5fe97d7868b78d4230c929851dab81"} Oct 30 20:01:46.950079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.949758 4990 scope.go:122] "RemoveContainer" containerID="7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b" Oct 30 20:01:46.960216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.960189 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.961786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.961758 4990 scope.go:122] "RemoveContainer" containerID="7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b" Oct 30 20:01:46.962114 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:46.962075 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b\": container with ID starting with 7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b not found: ID does not exist" containerID="7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b" Oct 30 20:01:46.962183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:46.962128 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b"} err="failed to get container status \"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b\": rpc error: code = NotFound desc = could not find container \"7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b\": container with ID starting with 7b5e95196502c25ea8de5f7d957de78b39ea25d46f569a8f98472d742563d53b not found: ID does not exist" Oct 30 20:01:47.001244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.001151 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 20:01:47.007766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.007731 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2afc11-555679777f-h7phc"] Oct 30 20:01:47.324599 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.324559 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.516290 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.516263 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 20:01:47.598165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598059 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598178 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598202 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598224 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598292 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598387 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598391 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:47.598583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598509 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:47.598583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598556 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598566 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:47.598869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598596 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598647 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\") pod \"ea654294-9e08-4a8c-9b6a-f33929f729a1\" (UID: \"ea654294-9e08-4a8c-9b6a-f33929f729a1\") " Oct 30 20:01:47.598869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598811 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:47.598869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.598818 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:47.599298 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.599277 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.599401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.599302 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.599401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.599317 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.599401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.599331 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea654294-9e08-4a8c-9b6a-f33929f729a1-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.599401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.599346 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.601578 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.601543 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:47.601702 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.601591 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:47.601702 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.601647 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "kube-api-access-pxfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:47.601921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.601904 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy" pod "ea654294-9e08-4a8c-9b6a-f33929f729a1" (UID: "ea654294-9e08-4a8c-9b6a-f33929f729a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:47.701042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.700575 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.701042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.700621 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxfks\" (UniqueName: \"kubernetes.io/projected/ea654294-9e08-4a8c-9b6a-f33929f729a1-kube-api-access-pxfks\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.701042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.700636 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.701042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.700649 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea654294-9e08-4a8c-9b6a-f33929f729a1-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.959164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.959064 4990 generic.go:356] "Generic (PLEG): container finished" podID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerID="79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657" exitCode=0 Oct 30 20:01:47.959164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.959110 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" event={"ID":"ea654294-9e08-4a8c-9b6a-f33929f729a1","Type":"ContainerDied","Data":"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657"} Oct 30 20:01:47.959738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.959172 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" event={"ID":"ea654294-9e08-4a8c-9b6a-f33929f729a1","Type":"ContainerDied","Data":"190d0960aec13550ade54a4e5350cd49c7ab510d5fa3d8cfa78902b38cc0cf23"} Oct 30 20:01:47.959738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.959128 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj" Oct 30 20:01:47.959738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.959196 4990 scope.go:122] "RemoveContainer" containerID="79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657" Oct 30 20:01:47.971963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.971937 4990 scope.go:122] "RemoveContainer" containerID="79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657" Oct 30 20:01:47.972383 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:01:47.972361 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657\": container with ID starting with 79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657 not found: ID does not exist" containerID="79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657" Oct 30 20:01:47.972460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:47.972391 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657"} err="failed to get container status \"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657\": rpc error: code = NotFound desc = could not find container \"79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657\": container with ID starting with 79a4a5491f7f53242201b30de08985236370238cbd6d1086c19ee1c714ed6657 not found: ID does not exist" Oct 30 20:01:48.006032 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.005991 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 20:01:48.015765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.015718 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-9jnqj"] Oct 30 20:01:48.418168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.418133 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c9e9db-10cd-423e-9364-115bc36770a6" path="/var/lib/kubelet/pods/49c9e9db-10cd-423e-9364-115bc36770a6/volumes" Oct 30 20:01:48.418585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.418571 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c170c74-2736-4d97-8b7d-c651fa404435" path="/var/lib/kubelet/pods/7c170c74-2736-4d97-8b7d-c651fa404435/volumes" Oct 30 20:01:48.419021 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.419008 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" path="/var/lib/kubelet/pods/ea654294-9e08-4a8c-9b6a-f33929f729a1/volumes" Oct 30 20:01:48.892630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:48.892585 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.469127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469071 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk"] Oct 30 20:01:50.469711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469694 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.469711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469711 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469721 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469726 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469735 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8e8bf6bf-b063-416d-923b-bcf21e98f692" containerName="payload-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469740 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8bf6bf-b063-416d-923b-bcf21e98f692" containerName="payload-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469752 4990 cpu_manager.go:414] "containerMap: removing container" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469758 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469765 4990 cpu_manager.go:414] "containerMap: removing container" podUID="49c9e9db-10cd-423e-9364-115bc36770a6" containerName="revoke-keys" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469769 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c9e9db-10cd-423e-9364-115bc36770a6" containerName="revoke-keys" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469785 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8049c7ef-e6a3-4095-82c1-dee826dbcd59" containerName="revoke-keys" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469789 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8049c7ef-e6a3-4095-82c1-dee826dbcd59" containerName="revoke-keys" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469798 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469802 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469815 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7c170c74-2736-4d97-8b7d-c651fa404435" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469820 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c170c74-2736-4d97-8b7d-c651fa404435" containerName="payload-pre-processing" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469831 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.469856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469836 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469924 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c170c74-2736-4d97-8b7d-c651fa404435" containerName="payload-pre-processing" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469934 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469944 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d691bb5f-4b13-4d10-9349-c4666c2c0f26" containerName="istio-proxy" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469950 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6fb294-f2d4-4061-b0bc-3ff24ef42a2b" containerName="payload-pre-processing" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469958 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8bf6bf-b063-416d-923b-bcf21e98f692" containerName="payload-processing" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469964 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea654294-9e08-4a8c-9b6a-f33929f729a1" containerName="istio-proxy" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469970 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c9e9db-10cd-423e-9364-115bc36770a6" containerName="revoke-keys" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469977 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8049c7ef-e6a3-4095-82c1-dee826dbcd59" containerName="revoke-keys" Oct 30 20:01:50.470737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.469983 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b82cbfb-4630-44b0-83a9-a38a5d48ff73" containerName="payload-pre-processing" Oct 30 20:01:50.475214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.475186 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.491349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.491316 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk"] Oct 30 20:01:50.496517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.496484 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:50.496517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.496519 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:01:50.645126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.645043 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.645320 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.645219 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.645439 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.645414 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25sf\" (UniqueName: \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.746464 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.746374 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.746647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.746573 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25sf\" (UniqueName: \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.746718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.746665 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.747292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.747271 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.748614 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.748594 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.763962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.763932 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25sf\" (UniqueName: \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.790682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.790647 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:50.810899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.810863 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.950617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:50.950582 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.109739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:51.109708 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk"] Oct 30 20:01:51.113234 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:51.113203 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823653fc_b004_4216_a1a0_78a781ab6c49.slice/crio-99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2 WatchSource:0}: Error finding container 99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2: Status 404 returned error can't find the container with id 99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2 Oct 30 20:01:51.134171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:51.134130 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.007836 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:52.007792 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" event={"ID":"823653fc-b004-4216-a1a0-78a781ab6c49","Type":"ContainerStarted","Data":"75a60e5fda899f31f0f5517b6a88cb8e651059d37859dc4cbe01125bab31396c"} Oct 30 20:01:52.008289 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:52.007899 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" event={"ID":"823653fc-b004-4216-a1a0-78a781ab6c49","Type":"ContainerStarted","Data":"99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2"} Oct 30 20:01:52.044218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:52.044157 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" podStartSLOduration=2.044139935 podStartE2EDuration="2.044139935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:52.041750883 +0000 UTC m=+3419.753276248" watchObservedRunningTime="2026-10-30 20:01:52.044139935 +0000 UTC m=+3419.755665303" Oct 30 20:01:52.891857 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:52.891817 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:53.018142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.018102 4990 generic.go:356] "Generic (PLEG): container finished" podID="823653fc-b004-4216-a1a0-78a781ab6c49" containerID="75a60e5fda899f31f0f5517b6a88cb8e651059d37859dc4cbe01125bab31396c" exitCode=0 Oct 30 20:01:53.018542 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.018142 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" event={"ID":"823653fc-b004-4216-a1a0-78a781ab6c49","Type":"ContainerDied","Data":"75a60e5fda899f31f0f5517b6a88cb8e651059d37859dc4cbe01125bab31396c"} Oct 30 20:01:53.211577 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.211501 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx"] Oct 30 20:01:53.215826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.215807 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.233760 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.233730 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx"] Oct 30 20:01:53.385458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.385408 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftj9\" (UniqueName: \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.385658 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.385627 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.385766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.385751 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.487063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.486954 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.487277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.487068 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.487277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.487164 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftj9\" (UniqueName: \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.487770 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.487741 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.489647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.489622 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.503847 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.503819 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftj9\" (UniqueName: \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\") pod \"maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.530300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.530272 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:53.731469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.731426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:53.841046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.841007 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx"] Oct 30 20:01:53.842988 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:53.842956 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf91caee_1fbb_4535_a525_c4342973d8c3.slice/crio-816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f WatchSource:0}: Error finding container 816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f: Status 404 returned error can't find the container with id 816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f Oct 30 20:01:53.892543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:53.892501 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.028545 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.028508 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" event={"ID":"bf91caee-1fbb-4535-a525-c4342973d8c3","Type":"ContainerStarted","Data":"816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f"} Oct 30 20:01:54.086459 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.086410 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.253121 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.253079 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:54.304414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.304375 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\" (UniqueName: \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\") pod \"823653fc-b004-4216-a1a0-78a781ab6c49\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " Oct 30 20:01:54.304648 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.304630 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\") pod \"823653fc-b004-4216-a1a0-78a781ab6c49\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " Oct 30 20:01:54.304750 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.304692 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\") pod \"823653fc-b004-4216-a1a0-78a781ab6c49\" (UID: \"823653fc-b004-4216-a1a0-78a781ab6c49\") " Oct 30 20:01:54.305302 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.305254 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca" pod "823653fc-b004-4216-a1a0-78a781ab6c49" (UID: "823653fc-b004-4216-a1a0-78a781ab6c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:54.307334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.307303 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf" pod "823653fc-b004-4216-a1a0-78a781ab6c49" (UID: "823653fc-b004-4216-a1a0-78a781ab6c49"). InnerVolumeSpecName "kube-api-access-k25sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:54.307529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.307504 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca" pod "823653fc-b004-4216-a1a0-78a781ab6c49" (UID: "823653fc-b004-4216-a1a0-78a781ab6c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:54.407048 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.407010 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:54.407048 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.407048 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25sf\" (UniqueName: \"kubernetes.io/projected/823653fc-b004-4216-a1a0-78a781ab6c49-kube-api-access-k25sf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:54.407275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.407065 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823653fc-b004-4216-a1a0-78a781ab6c49-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:54.510237 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.510205 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn"] Oct 30 20:01:54.510824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.510810 4990 cpu_manager.go:414] "containerMap: removing container" podUID="823653fc-b004-4216-a1a0-78a781ab6c49" containerName="revoke-keys" Oct 30 20:01:54.510871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.510826 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823653fc-b004-4216-a1a0-78a781ab6c49" containerName="revoke-keys" Oct 30 20:01:54.510935 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.510925 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="823653fc-b004-4216-a1a0-78a781ab6c49" containerName="revoke-keys" Oct 30 20:01:54.514783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.514756 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.534656 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.534624 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn"] Oct 30 20:01:54.609763 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.609719 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4bj\" (UniqueName: \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.609973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.609805 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.609973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.609846 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.639205 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.639169 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.710971 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.710938 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4bj\" (UniqueName: \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.711225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.710994 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.711225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.711015 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.711902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.711872 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.713555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.713531 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.727772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.727742 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4bj\" (UniqueName: \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:54.830835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:54.830798 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:55.038937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.038904 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" Oct 30 20:01:55.038937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.038926 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk" event={"ID":"823653fc-b004-4216-a1a0-78a781ab6c49","Type":"ContainerDied","Data":"99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2"} Oct 30 20:01:55.039429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.038977 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f712d075f36761055a7e7edfb7c6e6493a0f325b9214a5cb75c6da87fb5ac2" Oct 30 20:01:55.051986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.050584 4990 generic.go:356] "Generic (PLEG): container finished" podID="bf91caee-1fbb-4535-a525-c4342973d8c3" containerID="61afc38526cff51a8fa9435de7d146dbb582472ab7cd2cac7c0cfe07adb0e655" exitCode=0 Oct 30 20:01:55.051986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.050735 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" event={"ID":"bf91caee-1fbb-4535-a525-c4342973d8c3","Type":"ContainerDied","Data":"61afc38526cff51a8fa9435de7d146dbb582472ab7cd2cac7c0cfe07adb0e655"} Oct 30 20:01:55.147329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:55.147303 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn"] Oct 30 20:01:55.148543 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:01:55.148511 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2cd01f_5e32_4e43_aa7a_1f8dd0b3f16a.slice/crio-db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf WatchSource:0}: Error finding container db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf: Status 404 returned error can't find the container with id db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf Oct 30 20:01:56.060343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.060305 4990 generic.go:356] "Generic (PLEG): container finished" podID="cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" containerID="4b1eaabb3ba05ae6ff206bf945d4eed9c81a766bafe8ce00f2980d2175d4d615" exitCode=0 Oct 30 20:01:56.060817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.060390 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" event={"ID":"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a","Type":"ContainerDied","Data":"4b1eaabb3ba05ae6ff206bf945d4eed9c81a766bafe8ce00f2980d2175d4d615"} Oct 30 20:01:56.060817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.060442 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" event={"ID":"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a","Type":"ContainerStarted","Data":"db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf"} Oct 30 20:01:56.221820 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.221796 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:56.332363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.332324 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\") pod \"bf91caee-1fbb-4535-a525-c4342973d8c3\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " Oct 30 20:01:56.332558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.332432 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\" (UniqueName: \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\") pod \"bf91caee-1fbb-4535-a525-c4342973d8c3\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " Oct 30 20:01:56.332558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.332472 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\") pod \"bf91caee-1fbb-4535-a525-c4342973d8c3\" (UID: \"bf91caee-1fbb-4535-a525-c4342973d8c3\") " Oct 30 20:01:56.332795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.332765 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca" pod "bf91caee-1fbb-4535-a525-c4342973d8c3" (UID: "bf91caee-1fbb-4535-a525-c4342973d8c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:56.332898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.332882 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.334563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.334525 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca" pod "bf91caee-1fbb-4535-a525-c4342973d8c3" (UID: "bf91caee-1fbb-4535-a525-c4342973d8c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:56.334684 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.334668 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9" pod "bf91caee-1fbb-4535-a525-c4342973d8c3" (UID: "bf91caee-1fbb-4535-a525-c4342973d8c3"). InnerVolumeSpecName "kube-api-access-xftj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:56.434299 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.434267 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftj9\" (UniqueName: \"kubernetes.io/projected/bf91caee-1fbb-4535-a525-c4342973d8c3-kube-api-access-xftj9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.434299 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.434296 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf91caee-1fbb-4535-a525-c4342973d8c3-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:56.544463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:56.538552 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.069807 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.069774 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" Oct 30 20:01:57.069807 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.069783 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx" event={"ID":"bf91caee-1fbb-4535-a525-c4342973d8c3","Type":"ContainerDied","Data":"816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f"} Oct 30 20:01:57.070307 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.069842 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816ab74b639bc7151fc5065232bdbf794538a5c3acc5df58f3425483999dcd0f" Oct 30 20:01:57.230605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.230574 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:57.347557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.347466 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\") pod \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " Oct 30 20:01:57.347744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.347575 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\") pod \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " Oct 30 20:01:57.347744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.347624 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\" (UniqueName: \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\") pod \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\" (UID: \"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a\") " Oct 30 20:01:57.347946 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.347920 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca" pod "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" (UID: "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:57.349719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.349685 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj" pod "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" (UID: "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a"). InnerVolumeSpecName "kube-api-access-nl4bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:57.349810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.349774 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca" pod "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" (UID: "cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:57.385480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.385444 4990 scope.go:122] "RemoveContainer" containerID="dc6d38827ae3ffb383ef21e3cb3dfcbd7593983f880b58a2d5cea62d9348aad8" Oct 30 20:01:57.397984 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.397961 4990 scope.go:122] "RemoveContainer" containerID="bae7aac276e7b8c2f329b2729e07da16d522eaf4a1b2882b6e2d321fdf3dc4e2" Oct 30 20:01:57.410498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.410469 4990 scope.go:122] "RemoveContainer" containerID="74626cbfeab4db39bc473b8fe0979384dfff262116f930c43681ae2c6513b826" Oct 30 20:01:57.420431 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.420413 4990 scope.go:122] "RemoveContainer" containerID="7a80e1c180c5513d74210c04cfe8a772d7559962c96a07e7ee74099ab57c2f38" Oct 30 20:01:57.430178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.430154 4990 scope.go:122] "RemoveContainer" containerID="1850af562e5fa039c061e8b8f958b2277361cea2533792ce6b407972a8cee272" Oct 30 20:01:57.440033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.440010 4990 scope.go:122] "RemoveContainer" containerID="03ea4a1dd558cd18028b962dc31e3dec4990b5cd7654d90fb16b67ddc3951e11" Oct 30 20:01:57.450256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.450226 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:57.450256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.450257 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:57.450466 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.450274 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl4bj\" (UniqueName: \"kubernetes.io/projected/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a-kube-api-access-nl4bj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:01:57.453416 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:57.453396 4990 scope.go:122] "RemoveContainer" containerID="9f7d82932e2989b0a3fa7076874267f2b5bc2614287ec656cc2328676c46603c" Oct 30 20:01:58.079544 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:58.079514 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" Oct 30 20:01:58.079544 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:58.079535 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn" event={"ID":"cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a","Type":"ContainerDied","Data":"db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf"} Oct 30 20:01:58.080007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:58.079589 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5b4fedb46121363da2814d577e259242530b27cde0abd7488838b0b878efcf" Oct 30 20:01:58.173139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:58.172277 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.712134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:59.712001 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.764164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:59.764129 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 20:01:59.764429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:01:59.764403 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" containerID="cri-o://eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02" gracePeriod=30 Oct 30 20:02:00.967409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:00.967376 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 20:02:00.999826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:00.999783 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 20:02:01.146257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:01.146226 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.276962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:01.276854 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.471025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:01.470986 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.270969 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.270935 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.423128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.423050 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.561362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.561328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.866963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.866876 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.985518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.985470 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2"] Oct 30 20:02:03.995126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:03.995056 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-315ea2-76cp2"] Oct 30 20:02:04.024787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.024747 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.266710 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.266573 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.402057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.402026 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8049c7ef-e6a3-4095-82c1-dee826dbcd59" path="/var/lib/kubelet/pods/8049c7ef-e6a3-4095-82c1-dee826dbcd59/volumes" Oct 30 20:02:04.549166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.549064 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.965589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.965554 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 20:02:04.965793 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:04.965623 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 20:02:05.094690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.094659 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 20:02:05.105102 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:05.105056 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:02:05.167985 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.163361 4990 generic.go:356] "Generic (PLEG): container finished" podID="d4dfc325-47ac-4183-9276-b398c5517c54" containerID="eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02" exitCode=0 Oct 30 20:02:05.167985 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.163605 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" event={"ID":"d4dfc325-47ac-4183-9276-b398c5517c54","Type":"ContainerDied","Data":"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02"} Oct 30 20:02:05.167985 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.163717 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" event={"ID":"d4dfc325-47ac-4183-9276-b398c5517c54","Type":"ContainerDied","Data":"c41ee14dc2f0273d4c23b81ed78ed11ee31cc583bbe86635579f3c5da8370733"} Oct 30 20:02:05.167985 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.163761 4990 scope.go:122] "RemoveContainer" containerID="eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02" Oct 30 20:02:05.167985 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.163950 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6" Oct 30 20:02:05.174764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.174727 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.174902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.174829 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.174987 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.174941 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.175052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.175028 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.175131 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.175059 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.175409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.175380 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:05.175689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.175666 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:05.176585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.176556 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.177051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.177020 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:05.177208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.177082 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.177666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.177643 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:05.177758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.177703 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.178019 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.177995 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:05.178115 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.178051 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\") pod \"d4dfc325-47ac-4183-9276-b398c5517c54\" (UID: \"d4dfc325-47ac-4183-9276-b398c5517c54\") " Oct 30 20:02:05.178479 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.178451 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:05.179060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179034 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.179181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179064 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dfc325-47ac-4183-9276-b398c5517c54-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.179181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179079 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.179181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179110 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.179181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179124 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.179181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.179137 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dfc325-47ac-4183-9276-b398c5517c54-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.185217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.185109 4990 scope.go:122] "RemoveContainer" containerID="eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02" Oct 30 20:02:05.185675 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:05.185643 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02\": container with ID starting with eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02 not found: ID does not exist" containerID="eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02" Oct 30 20:02:05.185774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.185686 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02"} err="failed to get container status \"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02\": rpc error: code = NotFound desc = could not find container \"eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02\": container with ID starting with eca04390fde9e238509bbd9af9260a00e511fb5381a8cd526ea6f4f1c6cd4b02 not found: ID does not exist" Oct 30 20:02:05.188531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.188503 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:05.190086 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.190061 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:05.191888 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.191863 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw" pod "d4dfc325-47ac-4183-9276-b398c5517c54" (UID: "d4dfc325-47ac-4183-9276-b398c5517c54"). InnerVolumeSpecName "kube-api-access-bjmdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:05.214839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.214809 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.282952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.281106 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmdw\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-kube-api-access-bjmdw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.282952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.281193 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dfc325-47ac-4183-9276-b398c5517c54-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.282952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.281214 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dfc325-47ac-4183-9276-b398c5517c54-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:05.363277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.363235 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.508734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.508699 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 20:02:05.518909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.518881 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5b5v6"] Oct 30 20:02:05.967947 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:05.967898 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" containerID="cri-o://b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb" gracePeriod=25 Oct 30 20:02:06.000772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:06.000730 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" containerID="cri-o://da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca" gracePeriod=25 Oct 30 20:02:06.059109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:06.059056 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.401663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:06.401619 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" path="/var/lib/kubelet/pods/d4dfc325-47ac-4183-9276-b398c5517c54/volumes" Oct 30 20:02:07.857282 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:07.857247 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 20:02:07.857788 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:07.857496 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" podUID="8b663277-a13d-4f75-a96e-2dde01e07d43" containerName="maas-api" containerID="cri-o://01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129" gracePeriod=30 Oct 30 20:02:07.964142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:07.964100 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.160943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.160914 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 20:02:08.191784 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.191745 4990 generic.go:356] "Generic (PLEG): container finished" podID="8b663277-a13d-4f75-a96e-2dde01e07d43" containerID="01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129" exitCode=0 Oct 30 20:02:08.191999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.191806 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" Oct 30 20:02:08.191999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.191836 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" event={"ID":"8b663277-a13d-4f75-a96e-2dde01e07d43","Type":"ContainerDied","Data":"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129"} Oct 30 20:02:08.191999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.191955 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg" event={"ID":"8b663277-a13d-4f75-a96e-2dde01e07d43","Type":"ContainerDied","Data":"a38cc568145aa3a5036a729a58b7154014c86f5ea3d71e4fd55c494fb2649b16"} Oct 30 20:02:08.191999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.191994 4990 scope.go:122] "RemoveContainer" containerID="01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129" Oct 30 20:02:08.205121 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.205081 4990 scope.go:122] "RemoveContainer" containerID="01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129" Oct 30 20:02:08.205409 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:08.205388 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129\": container with ID starting with 01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129 not found: ID does not exist" containerID="01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129" Oct 30 20:02:08.205497 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.205422 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129"} err="failed to get container status \"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129\": rpc error: code = NotFound desc = could not find container \"01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129\": container with ID starting with 01db485293355c03b97e155442d78fdaaca547f1cd75c5344e4286314615d129 not found: ID does not exist" Oct 30 20:02:08.346194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.339545 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\") pod \"8b663277-a13d-4f75-a96e-2dde01e07d43\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " Oct 30 20:02:08.346194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.339628 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\" (UniqueName: \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\") pod \"8b663277-a13d-4f75-a96e-2dde01e07d43\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " Oct 30 20:02:08.346194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.339714 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\") pod \"8b663277-a13d-4f75-a96e-2dde01e07d43\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " Oct 30 20:02:08.346194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.339802 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") pod \"8b663277-a13d-4f75-a96e-2dde01e07d43\" (UID: \"8b663277-a13d-4f75-a96e-2dde01e07d43\") " Oct 30 20:02:08.346194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.341775 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca" pod "8b663277-a13d-4f75-a96e-2dde01e07d43" (UID: "8b663277-a13d-4f75-a96e-2dde01e07d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:08.362907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.362705 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7" pod "8b663277-a13d-4f75-a96e-2dde01e07d43" (UID: "8b663277-a13d-4f75-a96e-2dde01e07d43"). InnerVolumeSpecName "kube-api-access-jlxc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:08.362907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.362814 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls" pod "8b663277-a13d-4f75-a96e-2dde01e07d43" (UID: "8b663277-a13d-4f75-a96e-2dde01e07d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:08.407308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.407014 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs" pod "8b663277-a13d-4f75-a96e-2dde01e07d43" (UID: "8b663277-a13d-4f75-a96e-2dde01e07d43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:08.435478 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.435445 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.442050 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.442023 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b663277-a13d-4f75-a96e-2dde01e07d43-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:08.442050 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.442049 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlxc7\" (UniqueName: \"kubernetes.io/projected/8b663277-a13d-4f75-a96e-2dde01e07d43-kube-api-access-jlxc7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:08.442225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.442061 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:08.442225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.442070 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b663277-a13d-4f75-a96e-2dde01e07d43-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:08.522768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.522688 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn"] Oct 30 20:02:08.526423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.526390 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b7ccc-btwxn"] Oct 30 20:02:08.529550 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.529526 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 20:02:08.550280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.550244 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b7ccc-66b77b57c5-6v5qg"] Oct 30 20:02:08.588030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:08.587991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.010426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:09.010391 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 20:02:09.010923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:09.010626 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" containerID="cri-o://c0ec7dca8f283d4a8f5bedaaa0a1d9cfaa63c567aa960b9faff565592288075c" gracePeriod=30 Oct 30 20:02:09.071531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:09.071497 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.338986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:09.338946 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:09.339201 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:09.338959 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:10.335732 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:10.335691 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:10.402335 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:10.402295 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b663277-a13d-4f75-a96e-2dde01e07d43" path="/var/lib/kubelet/pods/8b663277-a13d-4f75-a96e-2dde01e07d43/volumes" Oct 30 20:02:10.402840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:10.402822 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" path="/var/lib/kubelet/pods/cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a/volumes" Oct 30 20:02:10.997278 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:10.997236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.137440 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.137401 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.221331 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.221277 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 20:02:11.221878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.221843 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" containerID="cri-o://99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b" gracePeriod=30 Oct 30 20:02:11.300206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.300176 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.733027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.732945 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd"] Oct 30 20:02:11.734509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734470 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bf91caee-1fbb-4535-a525-c4342973d8c3" containerName="revoke-keys" Oct 30 20:02:11.734509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734511 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf91caee-1fbb-4535-a525-c4342973d8c3" containerName="revoke-keys" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734548 4990 cpu_manager.go:414] "containerMap: removing container" podUID="cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" containerName="revoke-keys" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734556 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" containerName="revoke-keys" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734584 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8b663277-a13d-4f75-a96e-2dde01e07d43" containerName="maas-api" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734593 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b663277-a13d-4f75-a96e-2dde01e07d43" containerName="maas-api" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734639 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" Oct 30 20:02:11.734727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.734647 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" Oct 30 20:02:11.735229 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.735208 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf91caee-1fbb-4535-a525-c4342973d8c3" containerName="revoke-keys" Oct 30 20:02:11.735342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.735242 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2cd01f-5e32-4e43-aa7a-1f8dd0b3f16a" containerName="revoke-keys" Oct 30 20:02:11.735342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.735261 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b663277-a13d-4f75-a96e-2dde01e07d43" containerName="maas-api" Oct 30 20:02:11.735342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.735284 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4dfc325-47ac-4183-9276-b398c5517c54" containerName="istio-proxy" Oct 30 20:02:11.745854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.745814 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.755956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.755929 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd"] Oct 30 20:02:11.759881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.759853 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:11.760074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.759947 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:02:11.781511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.781476 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.782427 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.781589 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9c29\" (UniqueName: \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.782427 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.781718 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.882782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.882747 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.882960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.882837 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.882960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.882922 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9c29\" (UniqueName: \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.883607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.883578 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.885345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.885322 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:11.899782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:11.899755 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9c29\" (UniqueName: \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\") pod \"maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:12.064139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:12.064077 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:12.086084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:12.081783 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.389885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:12.389846 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd"] Oct 30 20:02:12.391939 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:12.391902 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b27c14_8139_40e0_85ea_5528e0cb5bd2.slice/crio-f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884 WatchSource:0}: Error finding container f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884: Status 404 returned error can't find the container with id f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884 Oct 30 20:02:13.238046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:13.238014 4990 generic.go:356] "Generic (PLEG): container finished" podID="22b27c14-8139-40e0-85ea-5528e0cb5bd2" containerID="2c2d7255e8dd4eca316ef63ce5298d69c9da50c5ad33cbfcef19548f0c8068b4" exitCode=0 Oct 30 20:02:13.238461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:13.238109 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" event={"ID":"22b27c14-8139-40e0-85ea-5528e0cb5bd2","Type":"ContainerDied","Data":"2c2d7255e8dd4eca316ef63ce5298d69c9da50c5ad33cbfcef19548f0c8068b4"} Oct 30 20:02:13.238461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:13.238167 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" event={"ID":"22b27c14-8139-40e0-85ea-5528e0cb5bd2","Type":"ContainerStarted","Data":"f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884"} Oct 30 20:02:13.821327 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:13.821286 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.247524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.247489 4990 generic.go:356] "Generic (PLEG): container finished" podID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerID="c0ec7dca8f283d4a8f5bedaaa0a1d9cfaa63c567aa960b9faff565592288075c" exitCode=0 Oct 30 20:02:14.247937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.247558 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" event={"ID":"75374cca-d96f-4bf1-93dd-ce57969e9f75","Type":"ContainerDied","Data":"c0ec7dca8f283d4a8f5bedaaa0a1d9cfaa63c567aa960b9faff565592288075c"} Oct 30 20:02:14.283396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.283368 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 20:02:14.315778 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.315623 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.315778 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.315736 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.316024 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.315828 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.316024 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.315943 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.317308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.316947 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:14.317308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317062 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:14.317308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317276 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:14.317566 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317546 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.317963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317661 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.317963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317699 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.317963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317735 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.317963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.317809 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\") pod \"75374cca-d96f-4bf1-93dd-ce57969e9f75\" (UID: \"75374cca-d96f-4bf1-93dd-ce57969e9f75\") " Oct 30 20:02:14.320130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.318819 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:14.320130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.319007 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75374cca-d96f-4bf1-93dd-ce57969e9f75-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.320130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.319029 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.320130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.319043 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.320130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.319060 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.320668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.320636 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:14.320744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.320653 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:14.321667 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.321643 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:14.322511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.322478 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:14.330024 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.330002 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg" pod "75374cca-d96f-4bf1-93dd-ce57969e9f75" (UID: "75374cca-d96f-4bf1-93dd-ce57969e9f75"). InnerVolumeSpecName "kube-api-access-jvllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:14.389428 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.389357 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:14.419779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.419751 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\" (UniqueName: \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\") pod \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " Oct 30 20:02:14.419903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.419833 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\") pod \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " Oct 30 20:02:14.419903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.419891 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\") pod \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\" (UID: \"22b27c14-8139-40e0-85ea-5528e0cb5bd2\") " Oct 30 20:02:14.420225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420200 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca" pod "22b27c14-8139-40e0-85ea-5528e0cb5bd2" (UID: "22b27c14-8139-40e0-85ea-5528e0cb5bd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:14.420342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420260 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.420342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420273 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.420342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420282 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.420342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420290 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvllg\" (UniqueName: \"kubernetes.io/projected/75374cca-d96f-4bf1-93dd-ce57969e9f75-kube-api-access-jvllg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.420342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.420300 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75374cca-d96f-4bf1-93dd-ce57969e9f75-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.421891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.421873 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29" pod "22b27c14-8139-40e0-85ea-5528e0cb5bd2" (UID: "22b27c14-8139-40e0-85ea-5528e0cb5bd2"). InnerVolumeSpecName "kube-api-access-s9c29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:14.421960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.421913 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca" pod "22b27c14-8139-40e0-85ea-5528e0cb5bd2" (UID: "22b27c14-8139-40e0-85ea-5528e0cb5bd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:14.521585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.521548 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.521585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.521580 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9c29\" (UniqueName: \"kubernetes.io/projected/22b27c14-8139-40e0-85ea-5528e0cb5bd2-kube-api-access-s9c29\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.521585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.521590 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b27c14-8139-40e0-85ea-5528e0cb5bd2-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:14.946997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:14.946959 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.094972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.094936 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.228266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.228178 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 20:02:15.228482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.228446 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" containerID="cri-o://1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800" gracePeriod=30 Oct 30 20:02:15.257418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.257385 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" Oct 30 20:02:15.257858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.257416 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf" event={"ID":"75374cca-d96f-4bf1-93dd-ce57969e9f75","Type":"ContainerDied","Data":"dbc88d5064cb3f6f7c837b1004fbffad7450d644ea031919295df323417e48b5"} Oct 30 20:02:15.257858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.257481 4990 scope.go:122] "RemoveContainer" containerID="c0ec7dca8f283d4a8f5bedaaa0a1d9cfaa63c567aa960b9faff565592288075c" Oct 30 20:02:15.259363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.259285 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" event={"ID":"22b27c14-8139-40e0-85ea-5528e0cb5bd2","Type":"ContainerDied","Data":"f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884"} Oct 30 20:02:15.259363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.259302 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd" Oct 30 20:02:15.259363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.259337 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ff108059fbed52c382bfff6f2f04a0934412431dcdb68f78f814f08740d884" Oct 30 20:02:15.300794 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.300761 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 20:02:15.304203 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.304175 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-jghsf"] Oct 30 20:02:15.305845 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.305806 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.763114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:15.761445 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:16.370693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.370650 4990 patch_prober.go:29] interesting pod/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 20:02:16.371329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.370734 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 20:02:16.412132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.405543 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" path="/var/lib/kubelet/pods/75374cca-d96f-4bf1-93dd-ce57969e9f75/volumes" Oct 30 20:02:16.522528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.522503 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 20:02:16.663598 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663560 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.663781 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663661 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.663781 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663749 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.663909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663891 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.663948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663926 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.664006 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.663985 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.664054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664013 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.664133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.664133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664044 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.664265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664156 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\") pod \"9557ea28-d430-4305-b313-107486eb594b\" (UID: \"9557ea28-d430-4305-b313-107486eb594b\") " Oct 30 20:02:16.664333 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664309 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.664440 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664415 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.664716 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664687 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:16.664864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664746 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.664864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664767 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.664864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664777 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.665044 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.664891 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.666355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.666323 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:16.666468 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.666437 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:16.666548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.666528 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.666613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.666567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn" pod "9557ea28-d430-4305-b313-107486eb594b" (UID: "9557ea28-d430-4305-b313-107486eb594b"). InnerVolumeSpecName "kube-api-access-ht6qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765627 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6qn\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-kube-api-access-ht6qn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765660 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9557ea28-d430-4305-b313-107486eb594b-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765674 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9557ea28-d430-4305-b313-107486eb594b-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765685 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765696 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9557ea28-d430-4305-b313-107486eb594b-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:16.765722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:16.765707 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9557ea28-d430-4305-b313-107486eb594b-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:17.281531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.281495 4990 generic.go:356] "Generic (PLEG): container finished" podID="9557ea28-d430-4305-b313-107486eb594b" containerID="99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b" exitCode=0 Oct 30 20:02:17.281719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.281559 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" Oct 30 20:02:17.281719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.281582 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerDied","Data":"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b"} Oct 30 20:02:17.281719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.281679 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5" event={"ID":"9557ea28-d430-4305-b313-107486eb594b","Type":"ContainerDied","Data":"3633274f3e78b506515cd4fef29589f991a1a5e36e1cfb33d717454879cd3d89"} Oct 30 20:02:17.281719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.281709 4990 scope.go:122] "RemoveContainer" containerID="99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b" Oct 30 20:02:17.294731 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.294708 4990 scope.go:122] "RemoveContainer" containerID="9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081" Oct 30 20:02:17.308476 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.308455 4990 scope.go:122] "RemoveContainer" containerID="99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b" Oct 30 20:02:17.308801 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:17.308772 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b\": container with ID starting with 99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b not found: ID does not exist" containerID="99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b" Oct 30 20:02:17.308996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.308812 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b"} err="failed to get container status \"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b\": rpc error: code = NotFound desc = could not find container \"99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b\": container with ID starting with 99420ae3d7c2b8e2ba18a942f4bcb103c003201479cace294e63eb1ae1ebb42b not found: ID does not exist" Oct 30 20:02:17.308996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.308843 4990 scope.go:122] "RemoveContainer" containerID="9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081" Oct 30 20:02:17.309192 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:17.309170 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081\": container with ID starting with 9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081 not found: ID does not exist" containerID="9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081" Oct 30 20:02:17.309248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.309205 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081"} err="failed to get container status \"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081\": rpc error: code = NotFound desc = could not find container \"9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081\": container with ID starting with 9f48ffdcd500778c590047709ae905039d2ece7b0cccd1d718f863df3784a081 not found: ID does not exist" Oct 30 20:02:17.338073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.338044 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 20:02:17.347686 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.347648 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-315ea2-openshift-default-656c6ccd4f-b4kb5"] Oct 30 20:02:17.797637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.797596 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:17.987699 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.987661 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 20:02:17.987968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:17.987922 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" containerID="cri-o://83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10" gracePeriod=30 Oct 30 20:02:18.402501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:18.402463 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9557ea28-d430-4305-b313-107486eb594b" path="/var/lib/kubelet/pods/9557ea28-d430-4305-b313-107486eb594b/volumes" Oct 30 20:02:19.339645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:19.339610 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:19.340155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:19.339610 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:19.361675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:19.361628 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.521378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.521353 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 20:02:20.618880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.618846 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.618925 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.618997 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619037 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619086 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619228 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619317 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:20.619470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619374 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619402 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\") pod \"70937de3-8424-4604-8652-ff61aad4a9b4\" (UID: \"70937de3-8424-4604-8652-ff61aad4a9b4\") " Oct 30 20:02:20.619595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619517 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:20.619774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619743 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:20.619929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619774 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:20.620053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.619962 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:20.620053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.620043 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.620223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.620068 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.620223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.620085 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70937de3-8424-4604-8652-ff61aad4a9b4-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.620223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.620137 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.621257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.621235 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:20.621383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.621282 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:20.621425 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.621407 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "kube-api-access-pb9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:20.621723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.621695 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token" pod "70937de3-8424-4604-8652-ff61aad4a9b4" (UID: "70937de3-8424-4604-8652-ff61aad4a9b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:20.721350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.721306 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.721350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.721336 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70937de3-8424-4604-8652-ff61aad4a9b4-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.721350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.721346 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9rq\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-kube-api-access-pb9rq\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.721350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.721357 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70937de3-8424-4604-8652-ff61aad4a9b4-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.721740 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.721366 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70937de3-8424-4604-8652-ff61aad4a9b4-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:20.763787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:20.763701 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.226347 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.226303 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.306286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.306243 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk"] Oct 30 20:02:21.316189 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.316151 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-287ea2-pd4jk"] Oct 30 20:02:21.319913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.319886 4990 generic.go:356] "Generic (PLEG): container finished" podID="70937de3-8424-4604-8652-ff61aad4a9b4" containerID="1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800" exitCode=0 Oct 30 20:02:21.320057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.319953 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" Oct 30 20:02:21.320057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.319972 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" event={"ID":"70937de3-8424-4604-8652-ff61aad4a9b4","Type":"ContainerDied","Data":"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800"} Oct 30 20:02:21.320057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.320046 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g" event={"ID":"70937de3-8424-4604-8652-ff61aad4a9b4","Type":"ContainerDied","Data":"8afd50a91bb5eb5070b9af6ad38b1c4cd1b46796a9b9ab0f5a6855f363260105"} Oct 30 20:02:21.320186 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.320068 4990 scope.go:122] "RemoveContainer" containerID="1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800" Oct 30 20:02:21.331950 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.331933 4990 scope.go:122] "RemoveContainer" containerID="1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800" Oct 30 20:02:21.332265 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:21.332236 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800\": container with ID starting with 1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800 not found: ID does not exist" containerID="1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800" Oct 30 20:02:21.332367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.332271 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800"} err="failed to get container status \"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800\": rpc error: code = NotFound desc = could not find container \"1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800\": container with ID starting with 1d6696214e44e0b1c14e8d9e7f1c2aeeeda8656b36a3f4cf14c363cebd334800 not found: ID does not exist" Oct 30 20:02:21.372117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.372049 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 20:02:21.379311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.379284 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-9tz9g"] Oct 30 20:02:21.382367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.382338 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.825248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:21.825213 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.402793 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:22.402759 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" path="/var/lib/kubelet/pods/70937de3-8424-4604-8652-ff61aad4a9b4/volumes" Oct 30 20:02:22.403284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:22.403269 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823653fc-b004-4216-a1a0-78a781ab6c49" path="/var/lib/kubelet/pods/823653fc-b004-4216-a1a0-78a781ab6c49/volumes" Oct 30 20:02:23.078075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.077913 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.263296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.263236 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 20:02:23.340603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.340569 4990 generic.go:356] "Generic (PLEG): container finished" podID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerID="83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10" exitCode=0 Oct 30 20:02:23.340804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.340618 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" Oct 30 20:02:23.340804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.340648 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" event={"ID":"31f58063-a9e0-43bb-ad80-7a0b866c6bcd","Type":"ContainerDied","Data":"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10"} Oct 30 20:02:23.340804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.340704 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655" event={"ID":"31f58063-a9e0-43bb-ad80-7a0b866c6bcd","Type":"ContainerDied","Data":"b826e1a388689d39e732b82b2a0e77edb6d8ac0edb6bb5893d4917e51d63642e"} Oct 30 20:02:23.340804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.340732 4990 scope.go:122] "RemoveContainer" containerID="83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10" Oct 30 20:02:23.350739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.350716 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.350884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.350761 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.350884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.350797 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.350884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.350855 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.350972 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351046 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351213 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.351370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351347 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351419 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351453 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.351524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351466 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\") pod \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\" (UID: \"31f58063-a9e0-43bb-ad80-7a0b866c6bcd\") " Oct 30 20:02:23.351524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351472 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.351706 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351644 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.351944 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.351920 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:23.352051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.352034 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.352226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.352057 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.352226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.352067 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.352226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.352076 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.352226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.352084 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.353683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.353651 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:23.354455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.354396 4990 scope.go:122] "RemoveContainer" containerID="83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10" Oct 30 20:02:23.354778 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:23.354754 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10\": container with ID starting with 83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10 not found: ID does not exist" containerID="83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10" Oct 30 20:02:23.354862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.354770 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.354862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.354784 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10"} err="failed to get container status \"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10\": rpc error: code = NotFound desc = could not find container \"83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10\": container with ID starting with 83e0718da37f5ab0a6492cee55b1aed53369a69a34a425a2235b97e66f6c8e10 not found: ID does not exist" Oct 30 20:02:23.355496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.355469 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:23.355651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.355629 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2" pod "31f58063-a9e0-43bb-ad80-7a0b866c6bcd" (UID: "31f58063-a9e0-43bb-ad80-7a0b866c6bcd"). InnerVolumeSpecName "kube-api-access-nx8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:23.453418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.453383 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.453418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.453415 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.453637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.453427 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8q2\" (UniqueName: \"kubernetes.io/projected/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-kube-api-access-nx8q2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.453637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.453443 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f58063-a9e0-43bb-ad80-7a0b866c6bcd-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:23.684267 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.684235 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 20:02:23.693979 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:23.693948 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-5z655"] Oct 30 20:02:24.402929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:24.402894 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" path="/var/lib/kubelet/pods/31f58063-a9e0-43bb-ad80-7a0b866c6bcd/volumes" Oct 30 20:02:24.607564 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:24.605224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:25.755753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.755716 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4"] Oct 30 20:02:25.756296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756282 4990 cpu_manager.go:414] "containerMap: removing container" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" Oct 30 20:02:25.756352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756297 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" Oct 30 20:02:25.756352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756308 4990 cpu_manager.go:414] "containerMap: removing container" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" Oct 30 20:02:25.756352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756315 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" Oct 30 20:02:25.756352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756331 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:25.756352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756337 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756371 4990 cpu_manager.go:414] "containerMap: removing container" podUID="22b27c14-8139-40e0-85ea-5528e0cb5bd2" containerName="revoke-keys" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756376 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b27c14-8139-40e0-85ea-5528e0cb5bd2" containerName="revoke-keys" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756382 4990 cpu_manager.go:414] "containerMap: removing container" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756387 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756467 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756475 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b27c14-8139-40e0-85ea-5528e0cb5bd2" containerName="revoke-keys" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756480 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="75374cca-d96f-4bf1-93dd-ce57969e9f75" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756485 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="70937de3-8424-4604-8652-ff61aad4a9b4" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756491 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f58063-a9e0-43bb-ad80-7a0b866c6bcd" containerName="istio-proxy" Oct 30 20:02:25.756540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.756499 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:25.761370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.761352 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.779787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.779761 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4"] Oct 30 20:02:25.789266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.789236 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:25.789431 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.789270 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:02:25.884414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.884370 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tfx\" (UniqueName: \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.884616 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.884524 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.884616 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.884556 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.986522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.986483 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.986678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.986538 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.986783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.986763 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tfx\" (UniqueName: \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.987381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.987346 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:25.988960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:25.988936 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:26.004388 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.004349 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tfx\" (UniqueName: \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:26.076085 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.076049 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:26.138178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.138139 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.308673 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.308643 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 20:02:26.312292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.312267 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 20:02:26.347801 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.347745 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.370728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.370691 4990 generic.go:356] "Generic (PLEG): container finished" podID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerID="da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca" exitCode=0 Oct 30 20:02:26.370937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.370733 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" Oct 30 20:02:26.370937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.370774 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" event={"ID":"e6a9a932-101a-4cdd-bd86-759cb4892678","Type":"ContainerDied","Data":"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca"} Oct 30 20:02:26.370937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.370836 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb" event={"ID":"e6a9a932-101a-4cdd-bd86-759cb4892678","Type":"ContainerDied","Data":"7918afdec811076b84f6d645212624ec8c33a3754acea702997f5e5aab0d043d"} Oct 30 20:02:26.370937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.370871 4990 scope.go:122] "RemoveContainer" containerID="da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca" Oct 30 20:02:26.372739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.372719 4990 generic.go:356] "Generic (PLEG): container finished" podID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerID="b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb" exitCode=0 Oct 30 20:02:26.372877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.372761 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" event={"ID":"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0","Type":"ContainerDied","Data":"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb"} Oct 30 20:02:26.372877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.372791 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" event={"ID":"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0","Type":"ContainerDied","Data":"c0b005b4fcb78035d3f8ada687b82edde4564824ba5d9031b6fb8767ea327ebf"} Oct 30 20:02:26.372877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.372850 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq" Oct 30 20:02:26.383822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.383800 4990 scope.go:122] "RemoveContainer" containerID="da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca" Oct 30 20:02:26.384232 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:26.384201 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca\": container with ID starting with da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca not found: ID does not exist" containerID="da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca" Oct 30 20:02:26.384353 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.384248 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca"} err="failed to get container status \"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca\": rpc error: code = NotFound desc = could not find container \"da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca\": container with ID starting with da6b4107627bf6304d05b6a94f8c5659a900abd8410410b6e61a04b4ac15f5ca not found: ID does not exist" Oct 30 20:02:26.384353 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.384273 4990 scope.go:122] "RemoveContainer" containerID="b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb" Oct 30 20:02:26.401344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.401181 4990 scope.go:122] "RemoveContainer" containerID="b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb" Oct 30 20:02:26.401865 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:26.401789 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb\": container with ID starting with b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb not found: ID does not exist" containerID="b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb" Oct 30 20:02:26.401865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.401824 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb"} err="failed to get container status \"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb\": rpc error: code = NotFound desc = could not find container \"b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb\": container with ID starting with b5d0045655967e7de1efb47c92ff9d3138e89b7f611bda6dc78d3c3b1fbaedbb not found: ID does not exist" Oct 30 20:02:26.404870 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:26.404847 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a907cd5_1737_41b2_9870_c8c1659aa161.slice/crio-3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03 WatchSource:0}: Error finding container 3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03: Status 404 returned error can't find the container with id 3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03 Oct 30 20:02:26.410152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.410130 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4"] Oct 30 20:02:26.489473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.489435 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.499688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.499657 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\") pod \"e6a9a932-101a-4cdd-bd86-759cb4892678\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " Oct 30 20:02:26.502862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.502483 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume" pod "e6a9a932-101a-4cdd-bd86-759cb4892678" (UID: "e6a9a932-101a-4cdd-bd86-759cb4892678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:26.502862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.502554 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\") pod \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " Oct 30 20:02:26.523370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.502943 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\" (UniqueName: \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\") pod \"e6a9a932-101a-4cdd-bd86-759cb4892678\" (UID: \"e6a9a932-101a-4cdd-bd86-759cb4892678\") " Oct 30 20:02:26.523370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.502995 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\" (UniqueName: \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\") pod \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\" (UID: \"1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0\") " Oct 30 20:02:26.523370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.503730 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a9a932-101a-4cdd-bd86-759cb4892678-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.523370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.504437 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume" pod "1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" (UID: "1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:26.535126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.524277 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r" pod "1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" (UID: "1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0"). InnerVolumeSpecName "kube-api-access-dkb5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:26.535126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.524373 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf" pod "e6a9a932-101a-4cdd-bd86-759cb4892678" (UID: "e6a9a932-101a-4cdd-bd86-759cb4892678"). InnerVolumeSpecName "kube-api-access-6lpsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:26.605401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.605364 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpsf\" (UniqueName: \"kubernetes.io/projected/e6a9a932-101a-4cdd-bd86-759cb4892678-kube-api-access-6lpsf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.605401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.605395 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkb5r\" (UniqueName: \"kubernetes.io/projected/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-kube-api-access-dkb5r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.605401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.605407 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.726198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.726160 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 20:02:26.730566 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.730533 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-287ea2-b94f7fc6d-cdzvq"] Oct 30 20:02:26.755441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.755409 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 20:02:26.758672 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.758645 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-287ea2-69d574d695-jbwvb"] Oct 30 20:02:26.839278 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.839230 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.894715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.894620 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx"] Oct 30 20:02:26.897775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.897745 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fe3e5e-n22lx"] Oct 30 20:02:26.996475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:26.996438 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.382013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:27.381972 4990 generic.go:356] "Generic (PLEG): container finished" podID="8a907cd5-1737-41b2-9870-c8c1659aa161" containerID="8c27bb4e5f952da0ea78967e3952c35f66cdf39942b03860feba88810fce9380" exitCode=0 Oct 30 20:02:27.382187 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:27.382016 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" event={"ID":"8a907cd5-1737-41b2-9870-c8c1659aa161","Type":"ContainerDied","Data":"8c27bb4e5f952da0ea78967e3952c35f66cdf39942b03860feba88810fce9380"} Oct 30 20:02:27.382187 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:27.382141 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" event={"ID":"8a907cd5-1737-41b2-9870-c8c1659aa161","Type":"ContainerStarted","Data":"3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03"} Oct 30 20:02:28.312835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.312765 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.401373 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.401338 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" path="/var/lib/kubelet/pods/1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0/volumes" Oct 30 20:02:28.401821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.401805 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf91caee-1fbb-4535-a525-c4342973d8c3" path="/var/lib/kubelet/pods/bf91caee-1fbb-4535-a525-c4342973d8c3/volumes" Oct 30 20:02:28.402208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.402191 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" path="/var/lib/kubelet/pods/e6a9a932-101a-4cdd-bd86-759cb4892678/volumes" Oct 30 20:02:28.460834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.460798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.556972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.556942 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:28.589167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.584500 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 20:02:28.589167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.584763 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" containerID="cri-o://2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500" gracePeriod=30 Oct 30 20:02:28.635917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.635879 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\" (UniqueName: \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\") pod \"8a907cd5-1737-41b2-9870-c8c1659aa161\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " Oct 30 20:02:28.636152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.635978 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\") pod \"8a907cd5-1737-41b2-9870-c8c1659aa161\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " Oct 30 20:02:28.636263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.636240 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\") pod \"8a907cd5-1737-41b2-9870-c8c1659aa161\" (UID: \"8a907cd5-1737-41b2-9870-c8c1659aa161\") " Oct 30 20:02:28.636701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.636667 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca" pod "8a907cd5-1737-41b2-9870-c8c1659aa161" (UID: "8a907cd5-1737-41b2-9870-c8c1659aa161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:28.636833 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.636759 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:28.638235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.638199 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx" pod "8a907cd5-1737-41b2-9870-c8c1659aa161" (UID: "8a907cd5-1737-41b2-9870-c8c1659aa161"). InnerVolumeSpecName "kube-api-access-h6tfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:28.638235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.638210 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca" pod "8a907cd5-1737-41b2-9870-c8c1659aa161" (UID: "8a907cd5-1737-41b2-9870-c8c1659aa161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:28.657370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.657341 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.737528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.737489 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6tfx\" (UniqueName: \"kubernetes.io/projected/8a907cd5-1737-41b2-9870-c8c1659aa161-kube-api-access-h6tfx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:28.737528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.737522 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a907cd5-1737-41b2-9870-c8c1659aa161-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:28.793988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:28.793952 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.155930 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.155892 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.278875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.278839 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c"] Oct 30 20:02:29.279430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279414 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" Oct 30 20:02:29.279430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279430 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279442 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8a907cd5-1737-41b2-9870-c8c1659aa161" containerName="revoke-keys" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279447 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a907cd5-1737-41b2-9870-c8c1659aa161" containerName="revoke-keys" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279459 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279464 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279485 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:29.279555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279490 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9557ea28-d430-4305-b313-107486eb594b" containerName="istio-proxy" Oct 30 20:02:29.279758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279586 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f79183a-38ca-4be3-ae0d-b5ca9cb1cba0" containerName="payload-processing" Oct 30 20:02:29.279758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279594 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a907cd5-1737-41b2-9870-c8c1659aa161" containerName="revoke-keys" Oct 30 20:02:29.279758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.279601 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a9a932-101a-4cdd-bd86-759cb4892678" containerName="payload-pre-processing" Oct 30 20:02:29.283070 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.283050 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.301434 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.301410 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c"] Oct 30 20:02:29.344524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.344494 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.344980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.344542 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgbd8\" (UniqueName: \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.344980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.344612 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.400786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.400753 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" event={"ID":"8a907cd5-1737-41b2-9870-c8c1659aa161","Type":"ContainerDied","Data":"3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03"} Oct 30 20:02:29.400786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.400777 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4" Oct 30 20:02:29.400978 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.400800 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8ec482e5a165cfb4091cfe73ea9c1164bae2386db11287ad28d26cab5b5c03" Oct 30 20:02:29.446014 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.445984 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.446215 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.446036 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgbd8\" (UniqueName: \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.446215 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.446066 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.446782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.446758 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.448374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.448356 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.463119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.463074 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgbd8\" (UniqueName: \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.596777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.596741 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:29.923136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:29.923068 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c"] Oct 30 20:02:29.924652 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:29.924611 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a8ec3a_f449_4baa_93bb_edd31b72ce37.slice/crio-b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75 WatchSource:0}: Error finding container b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75: Status 404 returned error can't find the container with id b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75 Oct 30 20:02:30.409757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:30.409724 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" event={"ID":"09a8ec3a-f449-4baa-93bb-edd31b72ce37","Type":"ContainerStarted","Data":"c760cb7cd2b258e090d47ea6481c029a7f996fbb506693cc2d50e4fcf9071338"} Oct 30 20:02:30.410150 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:30.409778 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" event={"ID":"09a8ec3a-f449-4baa-93bb-edd31b72ce37","Type":"ContainerStarted","Data":"b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75"} Oct 30 20:02:30.455516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:30.455392 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" podStartSLOduration=1.455376927 podStartE2EDuration="1.455376927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:30.442539843 +0000 UTC m=+3458.154065203" watchObservedRunningTime="2026-10-30 20:02:30.455376927 +0000 UTC m=+3458.166902309" Oct 30 20:02:31.147267 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:31.147153 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.418966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:31.418881 4990 generic.go:356] "Generic (PLEG): container finished" podID="09a8ec3a-f449-4baa-93bb-edd31b72ce37" containerID="c760cb7cd2b258e090d47ea6481c029a7f996fbb506693cc2d50e4fcf9071338" exitCode=0 Oct 30 20:02:31.419382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:31.418971 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" event={"ID":"09a8ec3a-f449-4baa-93bb-edd31b72ce37","Type":"ContainerDied","Data":"c760cb7cd2b258e090d47ea6481c029a7f996fbb506693cc2d50e4fcf9071338"} Oct 30 20:02:32.597152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.597119 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:32.697355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.697257 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\") pod \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " Oct 30 20:02:32.697355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.697343 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\" (UniqueName: \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\") pod \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " Oct 30 20:02:32.697614 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.697472 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\") pod \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\" (UID: \"09a8ec3a-f449-4baa-93bb-edd31b72ce37\") " Oct 30 20:02:32.697854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.697825 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca" pod "09a8ec3a-f449-4baa-93bb-edd31b72ce37" (UID: "09a8ec3a-f449-4baa-93bb-edd31b72ce37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:32.698006 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.697969 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:32.700063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.700031 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca" pod "09a8ec3a-f449-4baa-93bb-edd31b72ce37" (UID: "09a8ec3a-f449-4baa-93bb-edd31b72ce37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:32.700202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.700134 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8" pod "09a8ec3a-f449-4baa-93bb-edd31b72ce37" (UID: "09a8ec3a-f449-4baa-93bb-edd31b72ce37"). InnerVolumeSpecName "kube-api-access-jgbd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:32.799693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.799663 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a8ec3a-f449-4baa-93bb-edd31b72ce37-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:32.799693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.799691 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgbd8\" (UniqueName: \"kubernetes.io/projected/09a8ec3a-f449-4baa-93bb-edd31b72ce37-kube-api-access-jgbd8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:32.904908 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:32.904870 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.434886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.434845 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" event={"ID":"09a8ec3a-f449-4baa-93bb-edd31b72ce37","Type":"ContainerDied","Data":"b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75"} Oct 30 20:02:33.434886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.434869 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c" Oct 30 20:02:33.434886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.434896 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b31deeef57c1020bd80fedae6584c2445274167dc591c647c4bd0b92a8735c75" Oct 30 20:02:33.854293 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.854263 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 20:02:33.913971 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.913935 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914035 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914068 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914173 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914224 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914267 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914374 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914452 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914485 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\") pod \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\" (UID: \"be0a3c69-7a34-4427-b69d-33b2f0d11e35\") " Oct 30 20:02:33.914723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914649 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:33.914919 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.914895 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:33.915105 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.915069 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:33.915155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.915133 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:33.915207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.915160 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:33.915439 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.915415 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:33.915776 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.915749 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:33.917408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.917368 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "kube-api-access-fd5gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:33.917872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.917843 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:33.919831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.918552 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:33.919831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:33.919141 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy" pod "be0a3c69-7a34-4427-b69d-33b2f0d11e35" (UID: "be0a3c69-7a34-4427-b69d-33b2f0d11e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:34.016914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016817 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5gc\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-kube-api-access-fd5gc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.016914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016869 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.016914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016886 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.016914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016902 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.016914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016916 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.017246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016927 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.017246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.016941 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be0a3c69-7a34-4427-b69d-33b2f0d11e35-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:34.061764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.061728 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 20:02:34.066690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.066650 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 20:02:34.085384 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.085348 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 20:02:34.198339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.198297 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.443408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.443369 4990 generic.go:356] "Generic (PLEG): container finished" podID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerID="2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500" exitCode=0 Oct 30 20:02:34.443581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.443433 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" Oct 30 20:02:34.443581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.443444 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" event={"ID":"be0a3c69-7a34-4427-b69d-33b2f0d11e35","Type":"ContainerDied","Data":"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500"} Oct 30 20:02:34.443581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.443505 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s" event={"ID":"be0a3c69-7a34-4427-b69d-33b2f0d11e35","Type":"ContainerDied","Data":"2b7a0f7493527871480412915f6ec3af332b02760301b3558f3848e4e2d32c0d"} Oct 30 20:02:34.443581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.443527 4990 scope.go:122] "RemoveContainer" containerID="2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500" Oct 30 20:02:34.455159 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.455118 4990 scope.go:122] "RemoveContainer" containerID="2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500" Oct 30 20:02:34.458582 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:34.458498 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500\": container with ID starting with 2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500 not found: ID does not exist" containerID="2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500" Oct 30 20:02:34.458582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.458542 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500"} err="failed to get container status \"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500\": rpc error: code = NotFound desc = could not find container \"2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500\": container with ID starting with 2aa86d417eb15f33fc15e06ae6099546980b70159e0aef1fe8aad3d50b943500 not found: ID does not exist" Oct 30 20:02:34.460377 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.460328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.492829 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.492796 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 20:02:34.496749 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:34.496727 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lk25s"] Oct 30 20:02:35.097198 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:35.097157 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:02:36.014772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.014723 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.406503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.406465 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" path="/var/lib/kubelet/pods/be0a3c69-7a34-4427-b69d-33b2f0d11e35/volumes" Oct 30 20:02:36.499996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.499824 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.683530 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.683443 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.845172 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.845138 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.986839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:36.986748 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.443646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.443595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.571196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.571160 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:02:37.571399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.571375 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" podUID="845c4764-992b-445c-8efd-6198b7243572" containerName="payload-processing" containerID="cri-o://fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667" gracePeriod=30 Oct 30 20:02:37.599081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.599050 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:02:37.599296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.599274 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" podUID="1c0bf653-b968-4b38-a502-491d4a4c1a59" containerName="payload-pre-processing" containerID="cri-o://9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a" gracePeriod=30 Oct 30 20:02:37.612758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.606242 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.762452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.762236 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.963815 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.963748 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:02:37.985906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:37.985878 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:02:38.038207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.038171 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.063664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.063623 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\") pod \"1c0bf653-b968-4b38-a502-491d4a4c1a59\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " Oct 30 20:02:38.063865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.063773 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\" (UniqueName: \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\") pod \"845c4764-992b-445c-8efd-6198b7243572\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " Oct 30 20:02:38.063941 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.063916 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\") pod \"845c4764-992b-445c-8efd-6198b7243572\" (UID: \"845c4764-992b-445c-8efd-6198b7243572\") " Oct 30 20:02:38.063997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.063967 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\" (UniqueName: \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\") pod \"1c0bf653-b968-4b38-a502-491d4a4c1a59\" (UID: \"1c0bf653-b968-4b38-a502-491d4a4c1a59\") " Oct 30 20:02:38.064058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.064033 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume" pod "1c0bf653-b968-4b38-a502-491d4a4c1a59" (UID: "1c0bf653-b968-4b38-a502-491d4a4c1a59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:38.064349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.064325 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume" pod "845c4764-992b-445c-8efd-6198b7243572" (UID: "845c4764-992b-445c-8efd-6198b7243572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:38.064423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.064366 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0bf653-b968-4b38-a502-491d4a4c1a59-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:38.066195 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.066167 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n" pod "1c0bf653-b968-4b38-a502-491d4a4c1a59" (UID: "1c0bf653-b968-4b38-a502-491d4a4c1a59"). InnerVolumeSpecName "kube-api-access-qqq8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:38.066311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.066206 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc" pod "845c4764-992b-445c-8efd-6198b7243572" (UID: "845c4764-992b-445c-8efd-6198b7243572"). InnerVolumeSpecName "kube-api-access-gdclc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:38.115419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.115385 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd"] Oct 30 20:02:38.118693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.118664 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e430a0-kwmxd"] Oct 30 20:02:38.166109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.166058 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdclc\" (UniqueName: \"kubernetes.io/projected/845c4764-992b-445c-8efd-6198b7243572-kube-api-access-gdclc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:38.166109 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.166105 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c4764-992b-445c-8efd-6198b7243572-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:38.166338 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.166118 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq8n\" (UniqueName: \"kubernetes.io/projected/1c0bf653-b968-4b38-a502-491d4a4c1a59-kube-api-access-qqq8n\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:38.182053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.182016 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.401181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.401146 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b27c14-8139-40e0-85ea-5528e0cb5bd2" path="/var/lib/kubelet/pods/22b27c14-8139-40e0-85ea-5528e0cb5bd2/volumes" Oct 30 20:02:38.476137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.476071 4990 generic.go:356] "Generic (PLEG): container finished" podID="845c4764-992b-445c-8efd-6198b7243572" containerID="fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667" exitCode=0 Oct 30 20:02:38.476137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.476136 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" Oct 30 20:02:38.476696 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.476157 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" event={"ID":"845c4764-992b-445c-8efd-6198b7243572","Type":"ContainerDied","Data":"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667"} Oct 30 20:02:38.476696 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.476220 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj" event={"ID":"845c4764-992b-445c-8efd-6198b7243572","Type":"ContainerDied","Data":"ca549f94e5b14320f52151568f9041552becb29587785c6665625794726ef821"} Oct 30 20:02:38.476696 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.476254 4990 scope.go:122] "RemoveContainer" containerID="fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667" Oct 30 20:02:38.477877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.477850 4990 generic.go:356] "Generic (PLEG): container finished" podID="1c0bf653-b968-4b38-a502-491d4a4c1a59" containerID="9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a" exitCode=0 Oct 30 20:02:38.477991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.477893 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" Oct 30 20:02:38.477991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.477940 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" event={"ID":"1c0bf653-b968-4b38-a502-491d4a4c1a59","Type":"ContainerDied","Data":"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a"} Oct 30 20:02:38.477991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.477988 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w" event={"ID":"1c0bf653-b968-4b38-a502-491d4a4c1a59","Type":"ContainerDied","Data":"2a95838c86a157a49652af3d59acc21d7f91afc6dcf590bfcec6e23f72612937"} Oct 30 20:02:38.488046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.488029 4990 scope.go:122] "RemoveContainer" containerID="fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667" Oct 30 20:02:38.488368 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:38.488348 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667\": container with ID starting with fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667 not found: ID does not exist" containerID="fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667" Oct 30 20:02:38.488470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.488375 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667"} err="failed to get container status \"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667\": rpc error: code = NotFound desc = could not find container \"fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667\": container with ID starting with fdbb6666df2658d45046d220e1064eef95a5823665e6142b0f8c8764f68fc667 not found: ID does not exist" Oct 30 20:02:38.488470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.488391 4990 scope.go:122] "RemoveContainer" containerID="9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a" Oct 30 20:02:38.497989 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.497970 4990 scope.go:122] "RemoveContainer" containerID="9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a" Oct 30 20:02:38.498358 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:38.498326 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a\": container with ID starting with 9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a not found: ID does not exist" containerID="9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a" Oct 30 20:02:38.498437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.498363 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a"} err="failed to get container status \"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a\": rpc error: code = NotFound desc = could not find container \"9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a\": container with ID starting with 9fef743e5084b547eb66e668d663574ba3984cc80b16c709fedbd846c98ea50a not found: ID does not exist" Oct 30 20:02:38.521584 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.521558 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:02:38.533366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.533342 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-779c85db98-6jbnj"] Oct 30 20:02:38.559629 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.559607 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:02:38.564480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.564457 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-79b7cd8bbc-dpl2w"] Oct 30 20:02:38.696286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:38.696188 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.795143 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:39.795082 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 20:02:39.895917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:39.895881 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.044748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:40.044702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.402609 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:40.402574 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0bf653-b968-4b38-a502-491d4a4c1a59" path="/var/lib/kubelet/pods/1c0bf653-b968-4b38-a502-491d4a4c1a59/volumes" Oct 30 20:02:40.403005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:40.402991 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845c4764-992b-445c-8efd-6198b7243572" path="/var/lib/kubelet/pods/845c4764-992b-445c-8efd-6198b7243572/volumes" Oct 30 20:02:41.533127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:41.523841 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.140964 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.140806 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.390511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.390207 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.566344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.566307 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.656786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.656749 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 20:02:43.657138 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.657065 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" containerID="cri-o://43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec" gracePeriod=30 Oct 30 20:02:43.789154 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:43.769280 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.259941 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.259906 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw"] Oct 30 20:02:44.260498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260481 4990 cpu_manager.go:414] "containerMap: removing container" podUID="845c4764-992b-445c-8efd-6198b7243572" containerName="payload-processing" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260500 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c4764-992b-445c-8efd-6198b7243572" containerName="payload-processing" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260511 4990 cpu_manager.go:414] "containerMap: removing container" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260516 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260525 4990 cpu_manager.go:414] "containerMap: removing container" podUID="09a8ec3a-f449-4baa-93bb-edd31b72ce37" containerName="revoke-keys" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260530 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a8ec3a-f449-4baa-93bb-edd31b72ce37" containerName="revoke-keys" Oct 30 20:02:44.260549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260548 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1c0bf653-b968-4b38-a502-491d4a4c1a59" containerName="payload-pre-processing" Oct 30 20:02:44.260754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260553 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0bf653-b968-4b38-a502-491d4a4c1a59" containerName="payload-pre-processing" Oct 30 20:02:44.260754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260636 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0bf653-b968-4b38-a502-491d4a4c1a59" containerName="payload-pre-processing" Oct 30 20:02:44.260754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260651 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c4764-992b-445c-8efd-6198b7243572" containerName="payload-processing" Oct 30 20:02:44.260754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260657 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0a3c69-7a34-4427-b69d-33b2f0d11e35" containerName="istio-proxy" Oct 30 20:02:44.260754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.260663 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a8ec3a-f449-4baa-93bb-edd31b72ce37" containerName="revoke-keys" Oct 30 20:02:44.265232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.265209 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.281821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.281788 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw"] Oct 30 20:02:44.338777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.338730 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bpv\" (UniqueName: \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.338973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.338858 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.338973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.338930 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.374509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.374473 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:02:44.374697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.374519 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:44.439920 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.439882 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bpv\" (UniqueName: \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.440170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.439955 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.440170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.440001 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.440644 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.440612 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.442499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.442478 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.459285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.459258 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bpv\" (UniqueName: \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.559881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.559851 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj"] Oct 30 20:02:44.564038 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.564016 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.580270 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.580233 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:44.583833 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.583733 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj"] Oct 30 20:02:44.642723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.642679 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nb6\" (UniqueName: \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.642891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.642838 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.642988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.642966 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.744034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.743999 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nb6\" (UniqueName: \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.744225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.744057 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.744225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.744171 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.744762 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.744741 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.746372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.746347 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.761260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.761237 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nb6\" (UniqueName: \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\") pod \"maas-api-revoke-sub-2430d912-r5zcj\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.795917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.795880 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" containerID="cri-o://eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0" gracePeriod=25 Oct 30 20:02:44.879293 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.879212 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:44.899680 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:44.899651 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw"] Oct 30 20:02:44.901253 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:44.901222 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71944ad_4f6f_44b4_b5a1_a54b3b0c4bc8.slice/crio-a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38 WatchSource:0}: Error finding container a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38: Status 404 returned error can't find the container with id a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38 Oct 30 20:02:45.138272 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.138185 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.236500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.236029 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj"] Oct 30 20:02:45.239312 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:45.239278 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e1f9a2_a64e_4964_9245_1a8d7d58f82c.slice/crio-bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b WatchSource:0}: Error finding container bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b: Status 404 returned error can't find the container with id bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b Oct 30 20:02:45.292265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.292058 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.457441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.457359 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.539457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.539410 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" event={"ID":"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8","Type":"ContainerStarted","Data":"17c22b4ae2c50c4893eac2cc5ee1d8d7dd24b884c5f830df0043baee9682fdde"} Oct 30 20:02:45.539643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.539488 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" event={"ID":"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8","Type":"ContainerStarted","Data":"a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38"} Oct 30 20:02:45.541153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.541122 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" event={"ID":"46e1f9a2-a64e-4964-9245-1a8d7d58f82c","Type":"ContainerStarted","Data":"bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b"} Oct 30 20:02:45.553923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.553892 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 20:02:45.554158 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.554136 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" containerID="cri-o://e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a" gracePeriod=30 Oct 30 20:02:45.580032 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.579982 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" podStartSLOduration=1.5799652549999998 podStartE2EDuration="1.579965255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:45.578937243 +0000 UTC m=+3473.290462610" watchObservedRunningTime="2026-10-30 20:02:45.579965255 +0000 UTC m=+3473.291490621" Oct 30 20:02:45.634875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:45.634840 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.550595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:46.550561 4990 generic.go:356] "Generic (PLEG): container finished" podID="f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" containerID="17c22b4ae2c50c4893eac2cc5ee1d8d7dd24b884c5f830df0043baee9682fdde" exitCode=0 Oct 30 20:02:46.551046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:46.550599 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" event={"ID":"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8","Type":"ContainerDied","Data":"17c22b4ae2c50c4893eac2cc5ee1d8d7dd24b884c5f830df0043baee9682fdde"} Oct 30 20:02:46.552239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:46.552217 4990 generic.go:356] "Generic (PLEG): container finished" podID="46e1f9a2-a64e-4964-9245-1a8d7d58f82c" containerID="e731195a47100de4d09b14ddc095be9b7a9e53aa87f7f183025c7d2e3aea0eee" exitCode=0 Oct 30 20:02:46.552366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:46.552295 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" event={"ID":"46e1f9a2-a64e-4964-9245-1a8d7d58f82c","Type":"ContainerDied","Data":"e731195a47100de4d09b14ddc095be9b7a9e53aa87f7f183025c7d2e3aea0eee"} Oct 30 20:02:47.042919 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.042826 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.712780 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.712754 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:47.751591 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.751563 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:47.882523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882443 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\") pod \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " Oct 30 20:02:47.882523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882489 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\") pod \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " Oct 30 20:02:47.882734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882657 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\" (UniqueName: \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\") pod \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " Oct 30 20:02:47.882803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882786 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\") pod \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\" (UID: \"46e1f9a2-a64e-4964-9245-1a8d7d58f82c\") " Oct 30 20:02:47.882880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882828 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\" (UniqueName: \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\") pod \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " Oct 30 20:02:47.882880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882846 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca" pod "46e1f9a2-a64e-4964-9245-1a8d7d58f82c" (UID: "46e1f9a2-a64e-4964-9245-1a8d7d58f82c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:47.882880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.882862 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\") pod \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\" (UID: \"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8\") " Oct 30 20:02:47.883434 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.883407 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.884062 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.884034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca" pod "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" (UID: "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:47.884814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.884787 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca" pod "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" (UID: "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:47.885072 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.885051 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6" pod "46e1f9a2-a64e-4964-9245-1a8d7d58f82c" (UID: "46e1f9a2-a64e-4964-9245-1a8d7d58f82c"). InnerVolumeSpecName "kube-api-access-55nb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:47.885241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.885221 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca" pod "46e1f9a2-a64e-4964-9245-1a8d7d58f82c" (UID: "46e1f9a2-a64e-4964-9245-1a8d7d58f82c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:47.885697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.885667 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv" pod "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" (UID: "f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8"). InnerVolumeSpecName "kube-api-access-l9bpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:47.984536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.984497 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.984536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.984530 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bpv\" (UniqueName: \"kubernetes.io/projected/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-kube-api-access-l9bpv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.984536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.984542 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.984786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.984551 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:47.984786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:47.984560 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55nb6\" (UniqueName: \"kubernetes.io/projected/46e1f9a2-a64e-4964-9245-1a8d7d58f82c-kube-api-access-55nb6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:48.570121 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.570058 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" Oct 30 20:02:48.570121 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.570086 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw" event={"ID":"f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8","Type":"ContainerDied","Data":"a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38"} Oct 30 20:02:48.570382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.570168 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a39b63f6d6d4fe29731b69502c91a89b547515d99913652bb115aa304bd9bb38" Oct 30 20:02:48.571705 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.571682 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" Oct 30 20:02:48.571861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.571715 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj" event={"ID":"46e1f9a2-a64e-4964-9245-1a8d7d58f82c","Type":"ContainerDied","Data":"bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b"} Oct 30 20:02:48.571861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.571761 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4fccf2cb143c4e0ec13b88464849dfeb021760b731b3f4a0ff6e256506524b" Oct 30 20:02:48.730968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.730925 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:48.928416 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:48.928356 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 20:02:49.083951 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.083912 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" containerID="cri-o://4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70" gracePeriod=45 Oct 30 20:02:49.087611 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.087580 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" containerID="cri-o://0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396" gracePeriod=45 Oct 30 20:02:49.098083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098059 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098299 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098279 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098354 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098399 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098453 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098478 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098560 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098668 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098741 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\") pod \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\" (UID: \"d1a85259-5ff8-4caf-8d5b-195e5ead1504\") " Oct 30 20:02:49.098972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098944 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.098972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.098957 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.099204 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099183 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.099300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099278 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.099447 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099428 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.099518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099455 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.099518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099469 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.099632 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.099513 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:49.100610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.100588 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:49.100931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.100910 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "kube-api-access-cpz77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.101659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.101637 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.101727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.101683 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token" pod "d1a85259-5ff8-4caf-8d5b-195e5ead1504" (UID: "d1a85259-5ff8-4caf-8d5b-195e5ead1504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.105455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.105375 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" containerID="cri-o://5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd" gracePeriod=45 Oct 30 20:02:49.115917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.115022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.201460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201373 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.201460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201412 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.201460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201425 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.201460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201438 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.201460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201453 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpz77\" (UniqueName: \"kubernetes.io/projected/d1a85259-5ff8-4caf-8d5b-195e5ead1504-kube-api-access-cpz77\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.201851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.201467 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a85259-5ff8-4caf-8d5b-195e5ead1504-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.386903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.385588 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 20:02:49.390856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.390781 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 20:02:49.419430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.419402 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 20:02:49.507214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507125 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507232 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\" (UniqueName: \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507316 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\" (UniqueName: \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507348 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507424 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507458 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507562 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\" (UniqueName: \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507591 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.507761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507603 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\") pod \"0fc0847c-9475-4fd2-8184-1df73e421707\" (UID: \"0fc0847c-9475-4fd2-8184-1df73e421707\") " Oct 30 20:02:49.507761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507665 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507715 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507747 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\") pod \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\" (UID: \"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23\") " Oct 30 20:02:49.507975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.507832 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.508419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508382 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.508419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508406 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.508599 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508538 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.508663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508610 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.508864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508807 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.508976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.508956 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.510520 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.510491 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "kube-api-access-2qx7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.511517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.511494 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:49.511910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.511885 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.512032 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.512009 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.512463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.512443 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs" pod "0fc0847c-9475-4fd2-8184-1df73e421707" (UID: "0fc0847c-9475-4fd2-8184-1df73e421707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:49.512652 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.512632 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd" pod "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" (UID: "00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23"). InnerVolumeSpecName "kube-api-access-76nrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.582460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.582427 4990 generic.go:356] "Generic (PLEG): container finished" podID="0fc0847c-9475-4fd2-8184-1df73e421707" containerID="4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70" exitCode=0 Oct 30 20:02:49.582635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.582470 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" Oct 30 20:02:49.582635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.582503 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" event={"ID":"0fc0847c-9475-4fd2-8184-1df73e421707","Type":"ContainerDied","Data":"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70"} Oct 30 20:02:49.582635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.582544 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt" event={"ID":"0fc0847c-9475-4fd2-8184-1df73e421707","Type":"ContainerDied","Data":"0b14fb951d7942ad469308f46aa8b554316fba36efc1a587aed3638c4f5cc750"} Oct 30 20:02:49.582635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.582562 4990 scope.go:122] "RemoveContainer" containerID="4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70" Oct 30 20:02:49.584327 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.584301 4990 generic.go:356] "Generic (PLEG): container finished" podID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerID="0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396" exitCode=0 Oct 30 20:02:49.584470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.584348 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" Oct 30 20:02:49.584470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.584388 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" event={"ID":"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23","Type":"ContainerDied","Data":"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396"} Oct 30 20:02:49.584470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.584426 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h" event={"ID":"00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23","Type":"ContainerDied","Data":"28340b739ac929aebb461310c493bc989282cb93a11086ba02050fd93180b5d6"} Oct 30 20:02:49.586136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.586081 4990 generic.go:356] "Generic (PLEG): container finished" podID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerID="5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd" exitCode=0 Oct 30 20:02:49.586248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.586135 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" Oct 30 20:02:49.586248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.586134 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" event={"ID":"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2","Type":"ContainerDied","Data":"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd"} Oct 30 20:02:49.586377 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.586256 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk" event={"ID":"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2","Type":"ContainerDied","Data":"580079a5f39d5c0bdf6b7d87e84ae110b102444b67de2b3452af3c365c183fe2"} Oct 30 20:02:49.588627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.588397 4990 generic.go:356] "Generic (PLEG): container finished" podID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerID="43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec" exitCode=0 Oct 30 20:02:49.588627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.588451 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" Oct 30 20:02:49.588627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.588482 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerDied","Data":"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec"} Oct 30 20:02:49.588627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.588532 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l" event={"ID":"d1a85259-5ff8-4caf-8d5b-195e5ead1504","Type":"ContainerDied","Data":"4785806944c01b16357bf7a72054d26aff9df21db2c5fa3657e129c3467fb352"} Oct 30 20:02:49.596757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.596673 4990 scope.go:122] "RemoveContainer" containerID="4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70" Oct 30 20:02:49.597066 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:49.597038 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70\": container with ID starting with 4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70 not found: ID does not exist" containerID="4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70" Oct 30 20:02:49.597210 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.597075 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70"} err="failed to get container status \"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70\": rpc error: code = NotFound desc = could not find container \"4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70\": container with ID starting with 4a1b72cfcfbac05faf1c7651a9cf2c0d0a1b337e06a8d974daa77223c9ac1f70 not found: ID does not exist" Oct 30 20:02:49.597210 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.597128 4990 scope.go:122] "RemoveContainer" containerID="0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396" Oct 30 20:02:49.608301 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608283 4990 scope.go:122] "RemoveContainer" containerID="0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396" Oct 30 20:02:49.608527 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:49.608507 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396\": container with ID starting with 0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396 not found: ID does not exist" containerID="0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396" Oct 30 20:02:49.608579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608534 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396"} err="failed to get container status \"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396\": rpc error: code = NotFound desc = could not find container \"0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396\": container with ID starting with 0b09b004955f3562e9696d8ce88d1770a2ee6d5fe3fa743a144bb8c6faf48396 not found: ID does not exist" Oct 30 20:02:49.608579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608551 4990 scope.go:122] "RemoveContainer" containerID="5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd" Oct 30 20:02:49.608735 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608717 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.608824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608807 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.608876 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608861 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.608916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608896 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\" (UniqueName: \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.609005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.608990 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.609063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609037 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.609152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609135 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\") pod \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\" (UID: \"1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2\") " Oct 30 20:02:49.609199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609165 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.609285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609266 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.609605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609581 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609608 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609621 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609633 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609644 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609655 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609666 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609678 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qx7m\" (UniqueName: \"kubernetes.io/projected/0fc0847c-9475-4fd2-8184-1df73e421707-kube-api-access-2qx7m\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609693 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nrd\" (UniqueName: \"kubernetes.io/projected/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-kube-api-access-76nrd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609706 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609718 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0847c-9475-4fd2-8184-1df73e421707-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609732 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0847c-9475-4fd2-8184-1df73e421707-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.609744 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.610963 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.618060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.618040 4990 scope.go:122] "RemoveContainer" containerID="5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd" Oct 30 20:02:49.618400 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:49.618375 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd\": container with ID starting with 5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd not found: ID does not exist" containerID="5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd" Oct 30 20:02:49.618494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.618404 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd"} err="failed to get container status \"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd\": rpc error: code = NotFound desc = could not find container \"5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd\": container with ID starting with 5791b0b5e712cae164e5903a7885c89dbd48ca4280634d68313372d19d1a48cd not found: ID does not exist" Oct 30 20:02:49.618494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.618427 4990 scope.go:122] "RemoveContainer" containerID="43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec" Oct 30 20:02:49.623463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.623441 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "kube-api-access-6t7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.628475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.628455 4990 scope.go:122] "RemoveContainer" containerID="bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b" Oct 30 20:02:49.634409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.634386 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs" pod "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" (UID: "1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:49.637504 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.637488 4990 scope.go:122] "RemoveContainer" containerID="43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec" Oct 30 20:02:49.637756 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:49.637735 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec\": container with ID starting with 43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec not found: ID does not exist" containerID="43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec" Oct 30 20:02:49.637799 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.637764 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec"} err="failed to get container status \"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec\": rpc error: code = NotFound desc = could not find container \"43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec\": container with ID starting with 43f478532cd77bcef2a9018b91b735b4faba2c0c06995f077d56f57548796eec not found: ID does not exist" Oct 30 20:02:49.637799 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.637781 4990 scope.go:122] "RemoveContainer" containerID="bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b" Oct 30 20:02:49.638012 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:49.637992 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b\": container with ID starting with bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b not found: ID does not exist" containerID="bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b" Oct 30 20:02:49.638051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.638019 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b"} err="failed to get container status \"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b\": rpc error: code = NotFound desc = could not find container \"bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b\": container with ID starting with bfbad258ba0c098e31d46eb89d301197599b77bdd8912bf5157d6685131dae2b not found: ID does not exist" Oct 30 20:02:49.644254 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.644225 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 20:02:49.648581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.648558 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-2-w0-c5449c-kserve-fff65f6db-67sqt"] Oct 30 20:02:49.705404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.705371 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 20:02:49.710507 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.710463 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/distinct-w0-c5449c-kserve-5c769f964f-8tv8h"] Oct 30 20:02:49.710507 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.710508 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.710688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.710526 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7hd\" (UniqueName: \"kubernetes.io/projected/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-kube-api-access-6t7hd\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.710688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.710539 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.739863 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.739827 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 20:02:49.743674 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.743648 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fe3e5e-openshift-default-6f4b88c66f-52c9l"] Oct 30 20:02:49.928447 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.928416 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 20:02:49.932245 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:49.932216 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/simulator-w0-c5449c-kserve-6998d7c5d7-zz5sk"] Oct 30 20:02:50.401654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.401620 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" path="/var/lib/kubelet/pods/00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23/volumes" Oct 30 20:02:50.402101 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.402074 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" path="/var/lib/kubelet/pods/0fc0847c-9475-4fd2-8184-1df73e421707/volumes" Oct 30 20:02:50.402505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.402492 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" path="/var/lib/kubelet/pods/1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2/volumes" Oct 30 20:02:50.403248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.403232 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" path="/var/lib/kubelet/pods/d1a85259-5ff8-4caf-8d5b-195e5ead1504/volumes" Oct 30 20:02:50.746160 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.737970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:50.824674 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.824649 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 20:02:50.923666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923629 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.923855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923690 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.923855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923756 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.923855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923845 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923903 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923938 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.923972 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924020 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924084 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\") pod \"446334fe-4d24-4e9a-9968-b02208c0a647\" (UID: \"446334fe-4d24-4e9a-9968-b02208c0a647\") " Oct 30 20:02:50.924349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924136 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.924349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924221 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.924532 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924493 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.924754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924625 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.924754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924684 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.924754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924716 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.924754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924716 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.925080 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.924934 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:50.926488 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.926457 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:50.926769 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.926699 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.926769 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.926727 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.926920 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:50.926820 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56" pod "446334fe-4d24-4e9a-9968-b02208c0a647" (UID: "446334fe-4d24-4e9a-9968-b02208c0a647"). InnerVolumeSpecName "kube-api-access-ffm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:51.025843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025803 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446334fe-4d24-4e9a-9968-b02208c0a647-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.025843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025839 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446334fe-4d24-4e9a-9968-b02208c0a647-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.025843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025848 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.026110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025856 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.026110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025865 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446334fe-4d24-4e9a-9968-b02208c0a647-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.026110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.025873 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffm56\" (UniqueName: \"kubernetes.io/projected/446334fe-4d24-4e9a-9968-b02208c0a647-kube-api-access-ffm56\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:51.618744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.618708 4990 generic.go:356] "Generic (PLEG): container finished" podID="446334fe-4d24-4e9a-9968-b02208c0a647" containerID="e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a" exitCode=0 Oct 30 20:02:51.618929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.618765 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" Oct 30 20:02:51.618929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.618781 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerDied","Data":"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a"} Oct 30 20:02:51.618929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.618839 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7" event={"ID":"446334fe-4d24-4e9a-9968-b02208c0a647","Type":"ContainerDied","Data":"a31d68bddac37ebf19311fa38a0ae0f2707bbfcceaf84938dddf6fe7acacf92c"} Oct 30 20:02:51.618929 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.618866 4990 scope.go:122] "RemoveContainer" containerID="e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a" Oct 30 20:02:51.630593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.630573 4990 scope.go:122] "RemoveContainer" containerID="2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d" Oct 30 20:02:51.641230 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.641208 4990 scope.go:122] "RemoveContainer" containerID="e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a" Oct 30 20:02:51.641522 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:51.641497 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a\": container with ID starting with e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a not found: ID does not exist" containerID="e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a" Oct 30 20:02:51.641620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.641526 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a"} err="failed to get container status \"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a\": rpc error: code = NotFound desc = could not find container \"e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a\": container with ID starting with e5d918a9c29437c70219f926696a9ff78e6afd6205d68f0c8acef484f671719a not found: ID does not exist" Oct 30 20:02:51.641620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.641545 4990 scope.go:122] "RemoveContainer" containerID="2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d" Oct 30 20:02:51.641830 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:02:51.641814 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d\": container with ID starting with 2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d not found: ID does not exist" containerID="2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d" Oct 30 20:02:51.641881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.641836 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d"} err="failed to get container status \"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d\": rpc error: code = NotFound desc = could not find container \"2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d\": container with ID starting with 2aaa4c07931ba4032a019303e04d521903ced83cd2f7ad7da61a8e67de05683d not found: ID does not exist" Oct 30 20:02:51.673575 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.673543 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 20:02:51.679473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:51.679450 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e430a0-openshift-default-645446d55f-qdxq7"] Oct 30 20:02:52.401244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:52.401211 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" path="/var/lib/kubelet/pods/446334fe-4d24-4e9a-9968-b02208c0a647/volumes" Oct 30 20:02:52.518952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:52.518898 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.900572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:53.900533 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.396659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:54.396417 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.482963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:54.482924 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4"] Oct 30 20:02:54.492804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:54.492760 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9eb76-8tpc4"] Oct 30 20:02:54.541612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:54.541565 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.017425 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:55.017391 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.083240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083203 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw"] Oct 30 20:02:56.083697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083684 4990 cpu_manager.go:414] "containerMap: removing container" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.083754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083698 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.083754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083715 4990 cpu_manager.go:414] "containerMap: removing container" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.083754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083736 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.083754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083748 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:02:56.083754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083753 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083765 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" containerName="revoke-keys" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083770 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" containerName="revoke-keys" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083778 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083783 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083798 4990 cpu_manager.go:414] "containerMap: removing container" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083803 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083810 4990 cpu_manager.go:414] "containerMap: removing container" podUID="46e1f9a2-a64e-4964-9245-1a8d7d58f82c" containerName="revoke-keys" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083815 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e1f9a2-a64e-4964-9245-1a8d7d58f82c" containerName="revoke-keys" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083821 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083827 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083908 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:02:56.083916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083916 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083926 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="446334fe-4d24-4e9a-9968-b02208c0a647" containerName="istio-proxy" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083934 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" containerName="revoke-keys" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083939 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc0847c-9475-4fd2-8184-1df73e421707" containerName="main" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083946 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e0aa01-2f6b-4d1c-ae2e-9644e0d7bb23" containerName="main" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083951 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e1f9a2-a64e-4964-9245-1a8d7d58f82c" containerName="revoke-keys" Oct 30 20:02:56.084334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.083955 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d47e18d-4bff-42b8-8cbb-3cd5b22da1d2" containerName="main" Oct 30 20:02:56.089249 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.089227 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.113710 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.113676 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw"] Oct 30 20:02:56.161164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.161128 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:56.161340 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.161223 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:02:56.169418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.169392 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5sm8\" (UniqueName: \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.169496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.169431 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.169582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.169558 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.270721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.270681 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5sm8\" (UniqueName: \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.270934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.270735 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.270934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.270786 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.271421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.271399 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.273173 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.273147 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.282666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.282622 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.290445 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.290410 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5sm8\" (UniqueName: \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.402458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.402372 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a907cd5-1737-41b2-9870-c8c1659aa161" path="/var/lib/kubelet/pods/8a907cd5-1737-41b2-9870-c8c1659aa161/volumes" Oct 30 20:02:56.406557 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.406534 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:56.742252 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:56.742225 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw"] Oct 30 20:02:56.745424 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:02:56.745392 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52eb5d1_119f_421e_96ad_7baab155b733.slice/crio-b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e WatchSource:0}: Error finding container b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e: Status 404 returned error can't find the container with id b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e Oct 30 20:02:57.552715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:57.552686 4990 scope.go:122] "RemoveContainer" containerID="67e649be8a063f652e17ecaec9149bc4eec86257f10c38d8befceb3be5077432" Oct 30 20:02:57.562967 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:57.562945 4990 scope.go:122] "RemoveContainer" containerID="d3d75fb24ec5ad253a801022de04b3ff6751d5e40e1b1d35fd193f2488752880" Oct 30 20:02:57.669630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:57.669602 4990 generic.go:356] "Generic (PLEG): container finished" podID="e52eb5d1-119f-421e-96ad-7baab155b733" containerID="704e49f8acd478cc32221db8000679e140cc208c1836e57d2fb341d28d7d789c" exitCode=0 Oct 30 20:02:57.669748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:57.669687 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" event={"ID":"e52eb5d1-119f-421e-96ad-7baab155b733","Type":"ContainerDied","Data":"704e49f8acd478cc32221db8000679e140cc208c1836e57d2fb341d28d7d789c"} Oct 30 20:02:57.669748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:57.669741 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" event={"ID":"e52eb5d1-119f-421e-96ad-7baab155b733","Type":"ContainerStarted","Data":"b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e"} Oct 30 20:02:58.075108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.074974 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.730169 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.730134 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:58.844551 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.844525 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:58.901746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.901718 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\" (UniqueName: \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\") pod \"e52eb5d1-119f-421e-96ad-7baab155b733\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " Oct 30 20:02:58.901937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.901850 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\") pod \"e52eb5d1-119f-421e-96ad-7baab155b733\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " Oct 30 20:02:58.902006 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.901931 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\") pod \"e52eb5d1-119f-421e-96ad-7baab155b733\" (UID: \"e52eb5d1-119f-421e-96ad-7baab155b733\") " Oct 30 20:02:58.902401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.902370 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca" pod "e52eb5d1-119f-421e-96ad-7baab155b733" (UID: "e52eb5d1-119f-421e-96ad-7baab155b733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:58.903980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.903951 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8" pod "e52eb5d1-119f-421e-96ad-7baab155b733" (UID: "e52eb5d1-119f-421e-96ad-7baab155b733"). InnerVolumeSpecName "kube-api-access-h5sm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:58.904075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:58.903997 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca" pod "e52eb5d1-119f-421e-96ad-7baab155b733" (UID: "e52eb5d1-119f-421e-96ad-7baab155b733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:59.003664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.003586 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5sm8\" (UniqueName: \"kubernetes.io/projected/e52eb5d1-119f-421e-96ad-7baab155b733-kube-api-access-h5sm8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:59.003664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.003614 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:59.003664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.003624 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52eb5d1-119f-421e-96ad-7baab155b733-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:02:59.686396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.686356 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" Oct 30 20:02:59.686595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.686361 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw" event={"ID":"e52eb5d1-119f-421e-96ad-7baab155b733","Type":"ContainerDied","Data":"b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e"} Oct 30 20:02:59.686595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.686494 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b608b62ee7eaacbdca88c965de36f73d07655e894147207fe8e0c2e4f3ac965e" Oct 30 20:02:59.708714 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:02:59.708683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.285214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.285181 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.640915 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.640878 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.692279 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.692243 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 20:03:01.692515 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.692488 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" containerID="cri-o://97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047" gracePeriod=30 Oct 30 20:03:01.778737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.778698 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c"] Oct 30 20:03:01.783487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.783456 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5449c-tnh2c"] Oct 30 20:03:01.876467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:01.876424 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.164837 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.164753 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.292282 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292245 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp"] Oct 30 20:03:02.292748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292735 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:03:02.292798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292749 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:03:02.292798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292782 4990 cpu_manager.go:414] "containerMap: removing container" podUID="e52eb5d1-119f-421e-96ad-7baab155b733" containerName="revoke-keys" Oct 30 20:03:02.292798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292788 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52eb5d1-119f-421e-96ad-7baab155b733" containerName="revoke-keys" Oct 30 20:03:02.292911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292874 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52eb5d1-119f-421e-96ad-7baab155b733" containerName="revoke-keys" Oct 30 20:03:02.292911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.292886 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a85259-5ff8-4caf-8d5b-195e5ead1504" containerName="istio-proxy" Oct 30 20:03:02.296313 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.296293 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.317555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.317526 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp"] Oct 30 20:03:02.322889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.322040 4990 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:03:02.322889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.322085 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:03:02.325582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.325551 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:02.325709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.325613 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:03:02.337273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.337238 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.337448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.337292 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmt6r\" (UniqueName: \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.337595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.337565 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.339686 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.339631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.402800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.402754 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a8ec3a-f449-4baa-93bb-edd31b72ce37" path="/var/lib/kubelet/pods/09a8ec3a-f449-4baa-93bb-edd31b72ce37/volumes" Oct 30 20:03:02.438472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.438381 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.438472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.438450 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.438709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.438490 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmt6r\" (UniqueName: \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.439157 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.439132 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.440998 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.440969 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.456675 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.456639 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmt6r\" (UniqueName: \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\") pod \"maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.613501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.613465 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:02.915746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:02.915513 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp"] Oct 30 20:03:02.919208 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:03:02.919178 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda999cbef_fa78_412a_a6f8_2e8e771a056e.slice/crio-cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05 WatchSource:0}: Error finding container cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05: Status 404 returned error can't find the container with id cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05 Oct 30 20:03:03.392611 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.392572 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.551671 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.551631 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.727634 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.727597 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.727934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.727743 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" event={"ID":"a999cbef-fa78-412a-a6f8-2e8e771a056e","Type":"ContainerStarted","Data":"853a003a6feec157bffcc001444a9ae4bb08ee00c5cf1c58730a3ba68388143b"} Oct 30 20:03:03.728081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.727994 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" event={"ID":"a999cbef-fa78-412a-a6f8-2e8e771a056e","Type":"ContainerStarted","Data":"cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05"} Oct 30 20:03:03.765200 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.765149 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" podStartSLOduration=1.7651360010000001 podStartE2EDuration="1.765136001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:03.762128282 +0000 UTC m=+3491.473653645" watchObservedRunningTime="2026-10-30 20:03:03.765136001 +0000 UTC m=+3491.476661366" Oct 30 20:03:03.872253 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:03.872117 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.110143 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.110086 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.249235 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.249134 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.267143 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.267106 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw"] Oct 30 20:03:04.272477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.272435 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a76aa2-vq2cw"] Oct 30 20:03:04.393897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.393838 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.401514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.401483 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8" path="/var/lib/kubelet/pods/f71944ad-4f6f-44b4-b5a1-a54b3b0c4bc8/volumes" Oct 30 20:03:04.733108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.733061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.736856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.736822 4990 generic.go:356] "Generic (PLEG): container finished" podID="a999cbef-fa78-412a-a6f8-2e8e771a056e" containerID="853a003a6feec157bffcc001444a9ae4bb08ee00c5cf1c58730a3ba68388143b" exitCode=0 Oct 30 20:03:04.736977 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:04.736870 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" event={"ID":"a999cbef-fa78-412a-a6f8-2e8e771a056e","Type":"ContainerDied","Data":"853a003a6feec157bffcc001444a9ae4bb08ee00c5cf1c58730a3ba68388143b"} Oct 30 20:03:05.069488 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.069459 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 20:03:05.096725 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:05.096685 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:03:05.167890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.167850 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\" (UniqueName: \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\") pod \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " Oct 30 20:03:05.168059 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.168045 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\") pod \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\" (UID: \"42371951-c7d5-4fc2-a3a0-8a8af50a948c\") " Oct 30 20:03:05.168519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.168475 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume" pod "42371951-c7d5-4fc2-a3a0-8a8af50a948c" (UID: "42371951-c7d5-4fc2-a3a0-8a8af50a948c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:05.170417 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.170388 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg" pod "42371951-c7d5-4fc2-a3a0-8a8af50a948c" (UID: "42371951-c7d5-4fc2-a3a0-8a8af50a948c"). InnerVolumeSpecName "kube-api-access-jq5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:05.269534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.269489 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42371951-c7d5-4fc2-a3a0-8a8af50a948c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:05.269534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.269526 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5jg\" (UniqueName: \"kubernetes.io/projected/42371951-c7d5-4fc2-a3a0-8a8af50a948c-kube-api-access-jq5jg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:05.745153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.745084 4990 generic.go:356] "Generic (PLEG): container finished" podID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerID="eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0" exitCode=0 Oct 30 20:03:05.745153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.745141 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" Oct 30 20:03:05.745676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.745195 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" event={"ID":"42371951-c7d5-4fc2-a3a0-8a8af50a948c","Type":"ContainerDied","Data":"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0"} Oct 30 20:03:05.745676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.745253 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr" event={"ID":"42371951-c7d5-4fc2-a3a0-8a8af50a948c","Type":"ContainerDied","Data":"a3655f33c6e5a7ebbaa5a093ce3db29ae676c339ed362a82629ec8df2e2bd179"} Oct 30 20:03:05.745676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.745281 4990 scope.go:122] "RemoveContainer" containerID="eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0" Oct 30 20:03:05.755831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.755811 4990 scope.go:122] "RemoveContainer" containerID="eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0" Oct 30 20:03:05.756142 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:05.756121 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0\": container with ID starting with eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0 not found: ID does not exist" containerID="eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0" Oct 30 20:03:05.756188 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.756150 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0"} err="failed to get container status \"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0\": rpc error: code = NotFound desc = could not find container \"eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0\": container with ID starting with eb769480b6b3c32833da660ce287edefe37c3a613a55f157f0c59d68f7cecff0 not found: ID does not exist" Oct 30 20:03:05.792527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.792477 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 20:03:05.803500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.803467 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5449c-7d46ff99d6-fh2zr"] Oct 30 20:03:05.900435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.900405 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:05.978747 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.978709 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\" (UniqueName: \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\") pod \"a999cbef-fa78-412a-a6f8-2e8e771a056e\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " Oct 30 20:03:05.978976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.978829 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\") pod \"a999cbef-fa78-412a-a6f8-2e8e771a056e\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " Oct 30 20:03:05.978976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.978862 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\") pod \"a999cbef-fa78-412a-a6f8-2e8e771a056e\" (UID: \"a999cbef-fa78-412a-a6f8-2e8e771a056e\") " Oct 30 20:03:05.979310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.979274 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca" pod "a999cbef-fa78-412a-a6f8-2e8e771a056e" (UID: "a999cbef-fa78-412a-a6f8-2e8e771a056e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:05.981007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.980973 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r" pod "a999cbef-fa78-412a-a6f8-2e8e771a056e" (UID: "a999cbef-fa78-412a-a6f8-2e8e771a056e"). InnerVolumeSpecName "kube-api-access-rmt6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:05.981007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:05.980994 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca" pod "a999cbef-fa78-412a-a6f8-2e8e771a056e" (UID: "a999cbef-fa78-412a-a6f8-2e8e771a056e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:06.080665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.080626 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmt6r\" (UniqueName: \"kubernetes.io/projected/a999cbef-fa78-412a-a6f8-2e8e771a056e-kube-api-access-rmt6r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:06.080665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.080658 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:06.080665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.080668 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a999cbef-fa78-412a-a6f8-2e8e771a056e-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:06.116975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.116927 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.401987 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.401897 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" path="/var/lib/kubelet/pods/42371951-c7d5-4fc2-a3a0-8a8af50a948c/volumes" Oct 30 20:03:06.581151 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.581107 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 20:03:06.637258 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.637225 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 20:03:06.754543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.754516 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" Oct 30 20:03:06.754543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.754530 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp" event={"ID":"a999cbef-fa78-412a-a6f8-2e8e771a056e","Type":"ContainerDied","Data":"cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05"} Oct 30 20:03:06.755013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.754585 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac6b82ce39ab5b2202fcd5ce2cb7715a84065b1a031bfceac250b7db7ea8f05" Oct 30 20:03:06.781570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.781534 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.925680 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.925639 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.972191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:06.972122 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 20:03:07.094711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094661 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.094921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094776 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.094921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094812 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.094921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094844 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094923 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.094959 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095000 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095067 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095111 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\") pod \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\" (UID: \"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d\") " Oct 30 20:03:07.095141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095111 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:07.095572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095553 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.095818 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095797 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.096010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095912 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.096010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.095922 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.096276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.096250 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.098415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.098313 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.098415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.098374 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.098415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.098390 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:07.099009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.098937 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6" pod "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" (UID: "9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d"). InnerVolumeSpecName "kube-api-access-2tzf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.197356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197317 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzf6\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-kube-api-access-2tzf6\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197350 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197362 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197371 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197380 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197401 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197409 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.197605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.197417 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.218808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.218771 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:07.764176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.764139 4990 generic.go:356] "Generic (PLEG): container finished" podID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerID="97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047" exitCode=0 Oct 30 20:03:07.764618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.764206 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" Oct 30 20:03:07.764618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.764220 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" event={"ID":"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d","Type":"ContainerDied","Data":"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047"} Oct 30 20:03:07.764618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.764310 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5" event={"ID":"9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d","Type":"ContainerDied","Data":"2afcf85d1bf217a83abaec8becaa2b8743dde3a0f35316cbea604c061f4c79cd"} Oct 30 20:03:07.764618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.764340 4990 scope.go:122] "RemoveContainer" containerID="97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047" Oct 30 20:03:07.777841 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.777823 4990 scope.go:122] "RemoveContainer" containerID="97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047" Oct 30 20:03:07.778163 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:07.778142 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047\": container with ID starting with 97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047 not found: ID does not exist" containerID="97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047" Oct 30 20:03:07.778222 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.778171 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047"} err="failed to get container status \"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047\": rpc error: code = NotFound desc = could not find container \"97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047\": container with ID starting with 97fe59d5eade9f49c8f9cac44fd9f0ff04c7ed96f490c448b59b3ca237cdb047 not found: ID does not exist" Oct 30 20:03:07.813872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.813835 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 20:03:07.817670 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.817642 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-kqrz5"] Oct 30 20:03:07.834123 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:07.834082 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:08.400728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:08.400689 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" path="/var/lib/kubelet/pods/9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d/volumes" Oct 30 20:03:09.073195 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.073159 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.235825 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.235789 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.332952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.332867 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 20:03:09.333248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.333220 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" containerID="cri-o://4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83" gracePeriod=30 Oct 30 20:03:09.410567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.410384 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.556682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:09.556642 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.109279 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.109242 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.360107 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.359997 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.523875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.523835 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.584382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.583181 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" containerID="cri-o://afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b" gracePeriod=25 Oct 30 20:03:11.607443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.607408 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 20:03:11.607703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.607667 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" containerID="cri-o://b0fa8a3063ce52f7f5fc1016f59b2ecfa4b18da3deb1a54abbbbe1431eaaa8db" gracePeriod=30 Oct 30 20:03:11.638352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.638256 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" containerID="cri-o://599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8" gracePeriod=25 Oct 30 20:03:11.808635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.808589 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.957789 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:11.957693 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.168856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.168809 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8"] Oct 30 20:03:12.169381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169366 4990 cpu_manager.go:414] "containerMap: removing container" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" Oct 30 20:03:12.169446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169383 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" Oct 30 20:03:12.169446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169408 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" Oct 30 20:03:12.169446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169413 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" Oct 30 20:03:12.169446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169431 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a999cbef-fa78-412a-a6f8-2e8e771a056e" containerName="revoke-keys" Oct 30 20:03:12.169446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169436 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a999cbef-fa78-412a-a6f8-2e8e771a056e" containerName="revoke-keys" Oct 30 20:03:12.169713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169514 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="42371951-c7d5-4fc2-a3a0-8a8af50a948c" containerName="payload-processing" Oct 30 20:03:12.169713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169527 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6b9ecf-8d12-4f2c-abfe-3cd079fe1a1d" containerName="istio-proxy" Oct 30 20:03:12.169713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.169535 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a999cbef-fa78-412a-a6f8-2e8e771a056e" containerName="revoke-keys" Oct 30 20:03:12.174183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.174148 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.191801 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.191768 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8"] Oct 30 20:03:12.201579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.201553 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:03:12.201800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.201565 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:12.244786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.244689 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.266936 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.266905 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.267111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.266960 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.267208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.267085 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpjc\" (UniqueName: \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.368709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.368670 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpjc\" (UniqueName: \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.368901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.368799 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.368901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.368839 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.369494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.369462 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.371213 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.371188 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.386068 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.386043 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpjc\" (UniqueName: \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.439681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.439650 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.487940 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.487905 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:12.614718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.614681 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 20:03:12.793506 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.793037 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8"] Oct 30 20:03:12.796499 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:03:12.796470 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb80c70c_3b4e_4652_8492_fd9ef9500c83.slice/crio-79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427 WatchSource:0}: Error finding container 79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427: Status 404 returned error can't find the container with id 79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427 Oct 30 20:03:12.808980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.808949 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" event={"ID":"bb80c70c-3b4e-4652-8492-fd9ef9500c83","Type":"ContainerStarted","Data":"79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427"} Oct 30 20:03:12.813418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.813321 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.956840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:12.956798 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:13.107081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:13.107038 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:13.818826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:13.818789 4990 generic.go:356] "Generic (PLEG): container finished" podID="bb80c70c-3b4e-4652-8492-fd9ef9500c83" containerID="562731f4c20d183de1d802448c170649be001206d0a21e90e1d1d0e9a4ae6914" exitCode=0 Oct 30 20:03:13.819344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:13.818868 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" event={"ID":"bb80c70c-3b4e-4652-8492-fd9ef9500c83","Type":"ContainerDied","Data":"562731f4c20d183de1d802448c170649be001206d0a21e90e1d1d0e9a4ae6914"} Oct 30 20:03:14.494073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.494032 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.615610 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.615542 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 20:03:14.797123 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797068 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797132 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797180 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797205 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797250 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797292 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797324 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797394 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:14.797679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797405 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797469 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\") pod \"bc174af0-337d-42ce-992f-785837dce9bb\" (UID: \"bc174af0-337d-42ce-992f-785837dce9bb\") " Oct 30 20:03:14.797679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797624 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:14.797909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797813 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:14.797966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797879 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:14.798033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.797988 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc174af0-337d-42ce-992f-785837dce9bb-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.798033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.798012 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.798033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.798016 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:14.798033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.798028 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.798279 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.798041 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.799603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.799556 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:14.799894 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.799871 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:14.800027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.800006 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "kube-api-access-brrbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:14.800027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.799990 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token" pod "bc174af0-337d-42ce-992f-785837dce9bb" (UID: "bc174af0-337d-42ce-992f-785837dce9bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:14.828294 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.828257 4990 generic.go:356] "Generic (PLEG): container finished" podID="bc174af0-337d-42ce-992f-785837dce9bb" containerID="4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83" exitCode=0 Oct 30 20:03:14.828771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.828313 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" Oct 30 20:03:14.828771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.828337 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" event={"ID":"bc174af0-337d-42ce-992f-785837dce9bb","Type":"ContainerDied","Data":"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83"} Oct 30 20:03:14.828771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.828401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh" event={"ID":"bc174af0-337d-42ce-992f-785837dce9bb","Type":"ContainerDied","Data":"716a086ba7599def27d0d1292627250b263b90a18c74738f608a2ca0d6150b69"} Oct 30 20:03:14.828771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.828432 4990 scope.go:122] "RemoveContainer" containerID="4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83" Oct 30 20:03:14.841272 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.841251 4990 scope.go:122] "RemoveContainer" containerID="4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83" Oct 30 20:03:14.841562 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:14.841540 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83\": container with ID starting with 4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83 not found: ID does not exist" containerID="4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83" Oct 30 20:03:14.841643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.841573 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83"} err="failed to get container status \"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83\": rpc error: code = NotFound desc = could not find container \"4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83\": container with ID starting with 4e5b3746c97bef5c8a6062b4fb299b3a9fc328366ece0866edaf82ed931e9e83 not found: ID does not exist" Oct 30 20:03:14.876450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.876378 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 20:03:14.880451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.880424 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-ms7rh"] Oct 30 20:03:14.899086 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.899056 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.899086 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.899101 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrbs\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-kube-api-access-brrbs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.899300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.899122 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc174af0-337d-42ce-992f-785837dce9bb-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.899300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.899136 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc174af0-337d-42ce-992f-785837dce9bb-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.899300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.899147 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc174af0-337d-42ce-992f-785837dce9bb-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:14.966033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:14.966004 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:15.101581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.101546 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\") pod \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " Oct 30 20:03:15.101791 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.101605 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\" (UniqueName: \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\") pod \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " Oct 30 20:03:15.101791 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.101726 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\") pod \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\" (UID: \"bb80c70c-3b4e-4652-8492-fd9ef9500c83\") " Oct 30 20:03:15.102134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.102078 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca" pod "bb80c70c-3b4e-4652-8492-fd9ef9500c83" (UID: "bb80c70c-3b4e-4652-8492-fd9ef9500c83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:15.103714 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.103681 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca" pod "bb80c70c-3b4e-4652-8492-fd9ef9500c83" (UID: "bb80c70c-3b4e-4652-8492-fd9ef9500c83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:15.103805 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.103724 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc" pod "bb80c70c-3b4e-4652-8492-fd9ef9500c83" (UID: "bb80c70c-3b4e-4652-8492-fd9ef9500c83"). InnerVolumeSpecName "kube-api-access-4bpjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:15.203265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.203186 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:15.203265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.203217 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80c70c-3b4e-4652-8492-fd9ef9500c83-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:15.203265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.203228 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpjc\" (UniqueName: \"kubernetes.io/projected/bb80c70c-3b4e-4652-8492-fd9ef9500c83-kube-api-access-4bpjc\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:15.811841 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.811807 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.837469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.837384 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" Oct 30 20:03:15.837469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.837416 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8" event={"ID":"bb80c70c-3b4e-4652-8492-fd9ef9500c83","Type":"ContainerDied","Data":"79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427"} Oct 30 20:03:15.838004 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:15.837473 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b2ccfc7b236ec628d1a1dc8622a7a4fed5b26397b7d3594e85d535b1112427" Oct 30 20:03:16.135501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.135419 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd"] Oct 30 20:03:16.136257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136223 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" Oct 30 20:03:16.136257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136254 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" Oct 30 20:03:16.136510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136295 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bb80c70c-3b4e-4652-8492-fd9ef9500c83" containerName="revoke-keys" Oct 30 20:03:16.136510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136310 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb80c70c-3b4e-4652-8492-fd9ef9500c83" containerName="revoke-keys" Oct 30 20:03:16.136510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136501 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc174af0-337d-42ce-992f-785837dce9bb" containerName="istio-proxy" Oct 30 20:03:16.136689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.136520 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb80c70c-3b4e-4652-8492-fd9ef9500c83" containerName="revoke-keys" Oct 30 20:03:16.140223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.140201 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.158216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.158187 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd"] Oct 30 20:03:16.230939 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.230905 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:16.231164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.231005 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:03:16.292457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.292413 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.315651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.315617 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.315797 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.315771 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.315889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.315857 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zlr\" (UniqueName: \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.403138 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.403031 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc174af0-337d-42ce-992f-785837dce9bb" path="/var/lib/kubelet/pods/bc174af0-337d-42ce-992f-785837dce9bb/volumes" Oct 30 20:03:16.416499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.416470 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.416637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.416562 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.416637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.416593 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zlr\" (UniqueName: \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.417312 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.417286 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.418882 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.418860 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.435321 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.435296 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zlr\" (UniqueName: \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\") pod \"maas-api-revoke-sub-70f1b11d-w98sd\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.461331 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.461305 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:16.536875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.536844 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.537418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.537398 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw"] Oct 30 20:03:16.542018 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.541984 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-095f55-rpdsw"] Oct 30 20:03:16.726420 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.726380 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.778687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.778652 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd"] Oct 30 20:03:16.850356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.850309 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" event={"ID":"7d227471-b67a-4e3f-8225-f26fd36709a5","Type":"ContainerStarted","Data":"6108987e7e5e43a2a84d4ca250ee3ca26c8fb2b79e39b0fafd2e42a18118be1a"} Oct 30 20:03:16.852611 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.852579 4990 generic.go:356] "Generic (PLEG): container finished" podID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerID="b0fa8a3063ce52f7f5fc1016f59b2ecfa4b18da3deb1a54abbbbe1431eaaa8db" exitCode=0 Oct 30 20:03:16.852748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.852610 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerDied","Data":"b0fa8a3063ce52f7f5fc1016f59b2ecfa4b18da3deb1a54abbbbe1431eaaa8db"} Oct 30 20:03:16.852748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.852671 4990 scope.go:122] "RemoveContainer" containerID="1afaf109347cfc56597aa7cb458c9d92a79877f637efb5b5808ddb4c3034d2d4" Oct 30 20:03:16.884791 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:16.884772 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:03:17.025617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025585 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.025803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025638 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.025803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025703 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.025803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025727 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.025803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025763 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.026015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025793 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.026015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025888 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.026015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.025912 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026404 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026431 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026445 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026545 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\") pod \"bb15296f-4e20-44d2-a97e-bff775dc046f\" (UID: \"bb15296f-4e20-44d2-a97e-bff775dc046f\") " Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026824 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:17.026895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.026840 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:17.027354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.027125 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb15296f-4e20-44d2-a97e-bff775dc046f-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.027354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.027148 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.027354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.027165 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.027354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.027180 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.027354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.027193 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.028441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.028411 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:17.028600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.028581 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:17.028745 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.028724 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:17.028810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.028777 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w" pod "bb15296f-4e20-44d2-a97e-bff775dc046f" (UID: "bb15296f-4e20-44d2-a97e-bff775dc046f"). InnerVolumeSpecName "kube-api-access-j454w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:17.127943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.127905 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.127943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.127941 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.128221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.127953 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.128221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.127965 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j454w\" (UniqueName: \"kubernetes.io/projected/bb15296f-4e20-44d2-a97e-bff775dc046f-kube-api-access-j454w\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:17.615188 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.615145 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" containerID="cri-o://c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad" gracePeriod=25 Oct 30 20:03:17.862069 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.861982 4990 generic.go:356] "Generic (PLEG): container finished" podID="7d227471-b67a-4e3f-8225-f26fd36709a5" containerID="7ba26d37ff387a4ddda640a3dddb7b3359fd0184ae6f12f49dd3e32042478d97" exitCode=0 Oct 30 20:03:17.862512 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.862078 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" event={"ID":"7d227471-b67a-4e3f-8225-f26fd36709a5","Type":"ContainerDied","Data":"7ba26d37ff387a4ddda640a3dddb7b3359fd0184ae6f12f49dd3e32042478d97"} Oct 30 20:03:17.863651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.863623 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" Oct 30 20:03:17.863771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.863647 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5" event={"ID":"bb15296f-4e20-44d2-a97e-bff775dc046f","Type":"ContainerDied","Data":"5a11a1d1c7402670fea1900ca9130cf235b9377beb783f77751ca7284a44e806"} Oct 30 20:03:17.863771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.863703 4990 scope.go:122] "RemoveContainer" containerID="b0fa8a3063ce52f7f5fc1016f59b2ecfa4b18da3deb1a54abbbbe1431eaaa8db" Oct 30 20:03:17.957943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.957907 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 20:03:17.961438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:17.961412 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a76aa2-openshift-default-8588b5669-kzpg5"] Oct 30 20:03:18.367806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.367767 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:18.368033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.367767 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:18.410067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.410017 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" path="/var/lib/kubelet/pods/bb15296f-4e20-44d2-a97e-bff775dc046f/volumes" Oct 30 20:03:18.410723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.410691 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52eb5d1-119f-421e-96ad-7baab155b733" path="/var/lib/kubelet/pods/e52eb5d1-119f-421e-96ad-7baab155b733/volumes" Oct 30 20:03:18.459059 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.459026 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:18.919604 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:18.919555 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:19.015215 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.015191 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:19.066457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.066416 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:19.149396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.149303 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\") pod \"7d227471-b67a-4e3f-8225-f26fd36709a5\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " Oct 30 20:03:19.149396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.149372 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\") pod \"7d227471-b67a-4e3f-8225-f26fd36709a5\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " Oct 30 20:03:19.149638 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.149411 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\" (UniqueName: \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\") pod \"7d227471-b67a-4e3f-8225-f26fd36709a5\" (UID: \"7d227471-b67a-4e3f-8225-f26fd36709a5\") " Oct 30 20:03:19.149780 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.149758 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca" pod "7d227471-b67a-4e3f-8225-f26fd36709a5" (UID: "7d227471-b67a-4e3f-8225-f26fd36709a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:19.151574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.151541 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr" pod "7d227471-b67a-4e3f-8225-f26fd36709a5" (UID: "7d227471-b67a-4e3f-8225-f26fd36709a5"). InnerVolumeSpecName "kube-api-access-q5zlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:19.151574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.151545 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca" pod "7d227471-b67a-4e3f-8225-f26fd36709a5" (UID: "7d227471-b67a-4e3f-8225-f26fd36709a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:19.250825 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.250786 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5zlr\" (UniqueName: \"kubernetes.io/projected/7d227471-b67a-4e3f-8225-f26fd36709a5-kube-api-access-q5zlr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.250825 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.250816 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.250825 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.250826 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d227471-b67a-4e3f-8225-f26fd36709a5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:19.706102 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.706046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:19.884448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.884415 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" event={"ID":"7d227471-b67a-4e3f-8225-f26fd36709a5","Type":"ContainerDied","Data":"6108987e7e5e43a2a84d4ca250ee3ca26c8fb2b79e39b0fafd2e42a18118be1a"} Oct 30 20:03:19.884630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.884461 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd" Oct 30 20:03:19.884630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:19.884466 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6108987e7e5e43a2a84d4ca250ee3ca26c8fb2b79e39b0fafd2e42a18118be1a" Oct 30 20:03:20.004669 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:20.004577 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.666147 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:22.666115 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 20:03:22.697857 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:22.697823 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 20:03:22.760461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:22.760415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.913999 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:22.913961 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:23.312760 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:23.312719 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:24.592488 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:24.592451 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:24.898308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:24.898190 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.643481 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:26.643447 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:27.108083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:27.108045 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:27.666639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:27.666594 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" containerID="cri-o://e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78" gracePeriod=25 Oct 30 20:03:27.699001 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:27.698933 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" containerID="cri-o://2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61" gracePeriod=25 Oct 30 20:03:28.275033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:28.274989 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.367734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:28.367697 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:28.367943 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:28.367695 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:29.666466 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:29.666426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.181130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.181081 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.228347 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.228314 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8"] Oct 30 20:03:30.237806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.237775 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ec2fe7-mc5z8"] Oct 30 20:03:30.313447 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.313409 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.401281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.401247 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb80c70c-3b4e-4652-8492-fd9ef9500c83" path="/var/lib/kubelet/pods/bb80c70c-3b4e-4652-8492-fd9ef9500c83/volumes" Oct 30 20:03:30.754782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.754741 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:30.755254 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.754751 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:30.755474 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:30.755451 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.902417 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.902387 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 20:03:31.933903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.933877 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 20:03:31.981083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981045 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\") pod \"adfeddfa-aaf2-48bf-86c8-44554170b811\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " Oct 30 20:03:31.981311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981164 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\" (UniqueName: \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\") pod \"adfeddfa-aaf2-48bf-86c8-44554170b811\" (UID: \"adfeddfa-aaf2-48bf-86c8-44554170b811\") " Oct 30 20:03:31.981311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981203 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\" (UniqueName: \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\") pod \"16d153d4-e042-4b49-a64a-25cc4e542a7a\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " Oct 30 20:03:31.981311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981248 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\") pod \"16d153d4-e042-4b49-a64a-25cc4e542a7a\" (UID: \"16d153d4-e042-4b49-a64a-25cc4e542a7a\") " Oct 30 20:03:31.981553 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981528 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume" pod "adfeddfa-aaf2-48bf-86c8-44554170b811" (UID: "adfeddfa-aaf2-48bf-86c8-44554170b811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:31.981931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981850 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfeddfa-aaf2-48bf-86c8-44554170b811-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:31.981931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.981903 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume" pod "16d153d4-e042-4b49-a64a-25cc4e542a7a" (UID: "16d153d4-e042-4b49-a64a-25cc4e542a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:31.983852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.983358 4990 generic.go:356] "Generic (PLEG): container finished" podID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerID="599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8" exitCode=0 Oct 30 20:03:31.983852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.983394 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" event={"ID":"adfeddfa-aaf2-48bf-86c8-44554170b811","Type":"ContainerDied","Data":"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8"} Oct 30 20:03:31.983852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.983466 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" event={"ID":"adfeddfa-aaf2-48bf-86c8-44554170b811","Type":"ContainerDied","Data":"07631ee1aa492a8069bfa132b5b23499c20a48c4e463049b3ee20adf674c6c3b"} Oct 30 20:03:31.983852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.983501 4990 scope.go:122] "RemoveContainer" containerID="599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8" Oct 30 20:03:31.984222 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.983999 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2" pod "adfeddfa-aaf2-48bf-86c8-44554170b811" (UID: "adfeddfa-aaf2-48bf-86c8-44554170b811"). InnerVolumeSpecName "kube-api-access-h4bn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:31.984408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.984369 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw" pod "16d153d4-e042-4b49-a64a-25cc4e542a7a" (UID: "16d153d4-e042-4b49-a64a-25cc4e542a7a"). InnerVolumeSpecName "kube-api-access-rsqsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:31.985122 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.985075 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs" Oct 30 20:03:31.985417 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.985390 4990 generic.go:356] "Generic (PLEG): container finished" podID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerID="afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b" exitCode=0 Oct 30 20:03:31.985546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.985529 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" event={"ID":"16d153d4-e042-4b49-a64a-25cc4e542a7a","Type":"ContainerDied","Data":"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b"} Oct 30 20:03:31.985622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.985570 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" event={"ID":"16d153d4-e042-4b49-a64a-25cc4e542a7a","Type":"ContainerDied","Data":"b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795"} Oct 30 20:03:31.985622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.985574 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd" Oct 30 20:03:31.998356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.998331 4990 scope.go:122] "RemoveContainer" containerID="599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8" Oct 30 20:03:31.998945 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:31.998730 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8\": container with ID starting with 599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8 not found: ID does not exist" containerID="599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8" Oct 30 20:03:31.998945 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.998757 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8"} err="failed to get container status \"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8\": rpc error: code = NotFound desc = could not find container \"599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8\": container with ID starting with 599ee822adf2744a3a8d8c1544f501e767624c508951d5a698775f1f9b6eedd8 not found: ID does not exist" Oct 30 20:03:31.998945 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:31.998775 4990 scope.go:122] "RemoveContainer" containerID="afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b" Oct 30 20:03:32.011233 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.011212 4990 scope.go:122] "RemoveContainer" containerID="afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b" Oct 30 20:03:32.011525 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:32.011507 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b\": container with ID starting with afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b not found: ID does not exist" containerID="afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b" Oct 30 20:03:32.011596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.011532 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b"} err="failed to get container status \"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b\": rpc error: code = NotFound desc = could not find container \"afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b\": container with ID starting with afa49d32459f707493d6f81bf09855d40fca78db62b5ba4d8247384a23533e3b not found: ID does not exist" Oct 30 20:03:32.035684 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:32.035656 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d153d4_e042_4b49_a64a_25cc4e542a7a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d153d4_e042_4b49_a64a_25cc4e542a7a.slice/crio-b30c8311b527fd67c4536a220825242fa74ff260e756a3abfb4912c2a4255795\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfeddfa_aaf2_48bf_86c8_44554170b811.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:03:32.044703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.044678 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 20:03:32.047316 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.047295 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-095f55-5b97695c9-wsfgs"] Oct 30 20:03:32.079008 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.078980 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 20:03:32.083141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.083116 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4bn2\" (UniqueName: \"kubernetes.io/projected/adfeddfa-aaf2-48bf-86c8-44554170b811-kube-api-access-h4bn2\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:32.083248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.083143 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqsw\" (UniqueName: \"kubernetes.io/projected/16d153d4-e042-4b49-a64a-25cc4e542a7a-kube-api-access-rsqsw\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:32.083248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.083159 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d153d4-e042-4b49-a64a-25cc4e542a7a-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:32.084704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.084684 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-095f55-55bc548f7b-w8mcd"] Oct 30 20:03:32.109990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.109910 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.409269 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.409188 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" path="/var/lib/kubelet/pods/16d153d4-e042-4b49-a64a-25cc4e542a7a/volumes" Oct 30 20:03:32.409612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:32.409598 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" path="/var/lib/kubelet/pods/adfeddfa-aaf2-48bf-86c8-44554170b811/volumes" Oct 30 20:03:33.392211 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:33.392172 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.540451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:33.540403 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.757656 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:33.757563 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.908030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:33.907992 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.028206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:34.028118 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q"] Oct 30 20:03:34.037715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:34.037678 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65b8f637-92b2q"] Oct 30 20:03:34.400617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:34.400583 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777f2108-a13f-4a40-b736-41b50534b7c2" path="/var/lib/kubelet/pods/777f2108-a13f-4a40-b736-41b50534b7c2/volumes" Oct 30 20:03:34.592520 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:34.592482 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:35.096937 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:35.096900 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:03:35.368826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:35.368713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.906115 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:35.906052 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.983564 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:35.983521 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp"] Oct 30 20:03:35.992954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:35.992924 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9efc72-9mqcp"] Oct 30 20:03:36.056216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:36.056157 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.402382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:36.402347 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a999cbef-fa78-412a-a6f8-2e8e771a056e" path="/var/lib/kubelet/pods/a999cbef-fa78-412a-a6f8-2e8e771a056e/volumes" Oct 30 20:03:36.564257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:36.564216 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.536960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.536918 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.688770 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.688734 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.890424 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.890330 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.895527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.895502 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 20:03:37.941541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.941502 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\") pod \"67d56405-39e4-48bd-9623-bf65d5d279aa\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " Oct 30 20:03:37.941736 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.941589 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\" (UniqueName: \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\") pod \"67d56405-39e4-48bd-9623-bf65d5d279aa\" (UID: \"67d56405-39e4-48bd-9623-bf65d5d279aa\") " Oct 30 20:03:37.941887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.941858 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume" pod "67d56405-39e4-48bd-9623-bf65d5d279aa" (UID: "67d56405-39e4-48bd-9623-bf65d5d279aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:37.941956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.941941 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d56405-39e4-48bd-9623-bf65d5d279aa-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:37.944132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:37.944108 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h" pod "67d56405-39e4-48bd-9623-bf65d5d279aa" (UID: "67d56405-39e4-48bd-9623-bf65d5d279aa"). InnerVolumeSpecName "kube-api-access-zng6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:38.036631 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.036558 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.040661 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.040631 4990 generic.go:356] "Generic (PLEG): container finished" podID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerID="c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad" exitCode=0 Oct 30 20:03:38.040804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.040682 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" Oct 30 20:03:38.040804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.040711 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" event={"ID":"67d56405-39e4-48bd-9623-bf65d5d279aa","Type":"ContainerDied","Data":"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad"} Oct 30 20:03:38.040804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.040772 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp" event={"ID":"67d56405-39e4-48bd-9623-bf65d5d279aa","Type":"ContainerDied","Data":"aa9c4ad46df1b7987ec5731219b8f9804ad3f67fe89ca1521bc7b05f0cba6b75"} Oct 30 20:03:38.040804 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.040796 4990 scope.go:122] "RemoveContainer" containerID="c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad" Oct 30 20:03:38.042724 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.042705 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zng6h\" (UniqueName: \"kubernetes.io/projected/67d56405-39e4-48bd-9623-bf65d5d279aa-kube-api-access-zng6h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:38.055013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.054987 4990 scope.go:122] "RemoveContainer" containerID="c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad" Oct 30 20:03:38.055322 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:38.055304 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad\": container with ID starting with c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad not found: ID does not exist" containerID="c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad" Oct 30 20:03:38.055406 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.055330 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad"} err="failed to get container status \"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad\": rpc error: code = NotFound desc = could not find container \"c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad\": container with ID starting with c35946ff5d87da267309157193bb8a15c83ecbcc8a5957b1c82f9d01b08cf6ad not found: ID does not exist" Oct 30 20:03:38.090153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.090059 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 20:03:38.097682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.097654 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9efc72-79b9b67494-2dbgp"] Oct 30 20:03:38.401948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:38.401868 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" path="/var/lib/kubelet/pods/67d56405-39e4-48bd-9623-bf65d5d279aa/volumes" Oct 30 20:03:39.747219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:39.747179 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.754736 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:40.754648 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:40.755201 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:40.754653 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:41.431789 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:41.431740 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.041475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:43.041426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.273148 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:43.273116 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.460829 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:43.460785 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 20:03:43.461105 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:43.461065 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" containerID="cri-o://b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6" gracePeriod=30 Oct 30 20:03:43.682715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:43.682665 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.181695 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:44.181649 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.331456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:44.331421 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.762988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:44.762953 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 20:03:44.797884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:44.797845 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 20:03:44.964099 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:44.964048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.097792 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:45.097756 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.366822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:46.366782 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.607718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:46.607686 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.847806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:46.847760 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.999360 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:46.999325 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.963362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:47.963336 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 20:03:47.991583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:47.991560 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 20:03:48.049101 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.049057 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\") pod \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " Oct 30 20:03:48.049300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.049235 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\" (UniqueName: \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\") pod \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\" (UID: \"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa\") " Oct 30 20:03:48.049481 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.049456 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume" pod "dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" (UID: "dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:48.051278 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.051251 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j" pod "dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" (UID: "dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa"). InnerVolumeSpecName "kube-api-access-pl76j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:48.114160 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.114123 4990 generic.go:356] "Generic (PLEG): container finished" podID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerID="e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78" exitCode=0 Oct 30 20:03:48.114372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.114221 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" event={"ID":"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa","Type":"ContainerDied","Data":"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78"} Oct 30 20:03:48.114372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.114227 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" Oct 30 20:03:48.114372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.114261 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h" event={"ID":"dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa","Type":"ContainerDied","Data":"24d88508159ad7aaf9136b1a040ed462107ce2be8f4f824619a431ad42f3ba74"} Oct 30 20:03:48.114372 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.114292 4990 scope.go:122] "RemoveContainer" containerID="e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78" Oct 30 20:03:48.116057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.116030 4990 generic.go:356] "Generic (PLEG): container finished" podID="a6577b72-e951-4092-a434-648c4dec7b60" containerID="2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61" exitCode=0 Oct 30 20:03:48.116206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.116072 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" Oct 30 20:03:48.116206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.116073 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" event={"ID":"a6577b72-e951-4092-a434-648c4dec7b60","Type":"ContainerDied","Data":"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61"} Oct 30 20:03:48.116349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.116219 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j" event={"ID":"a6577b72-e951-4092-a434-648c4dec7b60","Type":"ContainerDied","Data":"ff4d1d287ee36e78263d99e0d90391070225acab4c1cadcaa0d64adaa70c3341"} Oct 30 20:03:48.125316 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.125298 4990 scope.go:122] "RemoveContainer" containerID="e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78" Oct 30 20:03:48.125597 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:48.125579 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78\": container with ID starting with e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78 not found: ID does not exist" containerID="e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78" Oct 30 20:03:48.125658 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.125605 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78"} err="failed to get container status \"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78\": rpc error: code = NotFound desc = could not find container \"e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78\": container with ID starting with e3276681c3767c28a2258e651ccc8a188c74d44d5c053f97faed61eec29b9f78 not found: ID does not exist" Oct 30 20:03:48.125658 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.125622 4990 scope.go:122] "RemoveContainer" containerID="2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61" Oct 30 20:03:48.134997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.134981 4990 scope.go:122] "RemoveContainer" containerID="2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61" Oct 30 20:03:48.135362 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:48.135342 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61\": container with ID starting with 2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61 not found: ID does not exist" containerID="2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61" Oct 30 20:03:48.135425 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.135369 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61"} err="failed to get container status \"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61\": rpc error: code = NotFound desc = could not find container \"2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61\": container with ID starting with 2b2678cac04e8b3d5a25015f336a221025258bb31830290f589a602b94a59e61 not found: ID does not exist" Oct 30 20:03:48.149903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.149880 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\" (UniqueName: \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\") pod \"a6577b72-e951-4092-a434-648c4dec7b60\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " Oct 30 20:03:48.150008 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.149955 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\") pod \"a6577b72-e951-4092-a434-648c4dec7b60\" (UID: \"a6577b72-e951-4092-a434-648c4dec7b60\") " Oct 30 20:03:48.150261 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.150245 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl76j\" (UniqueName: \"kubernetes.io/projected/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-kube-api-access-pl76j\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.150308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.150265 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.150371 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.150351 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume" pod "a6577b72-e951-4092-a434-648c4dec7b60" (UID: "a6577b72-e951-4092-a434-648c4dec7b60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:48.151960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.151941 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p" pod "a6577b72-e951-4092-a434-648c4dec7b60" (UID: "a6577b72-e951-4092-a434-648c4dec7b60"). InnerVolumeSpecName "kube-api-access-vnm6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:48.183325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.183291 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 20:03:48.197886 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.197809 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-ec2fe7-6895bc5946-77l6h"] Oct 30 20:03:48.251831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.251796 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnm6p\" (UniqueName: \"kubernetes.io/projected/a6577b72-e951-4092-a434-648c4dec7b60-kube-api-access-vnm6p\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.251831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.251826 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6577b72-e951-4092-a434-648c4dec7b60-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.401861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.401825 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" path="/var/lib/kubelet/pods/dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa/volumes" Oct 30 20:03:48.452456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.452377 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 20:03:48.469113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.464841 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-ec2fe7-5fdbd44cff-gvt8j"] Oct 30 20:03:48.723911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.723877 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.723911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.723904 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 20:03:48.856669 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856639 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856698 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856717 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856756 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856796 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856829 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.856875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856866 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.856903 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857008 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\") pod \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\" (UID: \"eba5e617-1e5c-47bc-94ed-0a9937f4c1da\") " Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857050 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857070 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857129 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.857273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857238 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:48.857561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857438 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.857561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857454 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.857561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857463 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.857561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857471 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.857802 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.857598 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.859395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.859370 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "kube-api-access-p89ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:48.859532 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.859460 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.859821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.859793 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:48.859925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.859859 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token" pod "eba5e617-1e5c-47bc-94ed-0a9937f4c1da" (UID: "eba5e617-1e5c-47bc-94ed-0a9937f4c1da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:48.874438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.874412 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 20:03:48.902010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.901978 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 20:03:48.959058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.958966 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.959058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.959017 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.959058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.959029 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.959058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.959043 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89ng\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-kube-api-access-p89ng\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:48.959058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:48.959057 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba5e617-1e5c-47bc-94ed-0a9937f4c1da-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:03:49.072924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.072877 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.124223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.124189 4990 generic.go:356] "Generic (PLEG): container finished" podID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerID="b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6" exitCode=0 Oct 30 20:03:49.124430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.124241 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" Oct 30 20:03:49.124430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.124269 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" event={"ID":"eba5e617-1e5c-47bc-94ed-0a9937f4c1da","Type":"ContainerDied","Data":"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6"} Oct 30 20:03:49.124430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.124331 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm" event={"ID":"eba5e617-1e5c-47bc-94ed-0a9937f4c1da","Type":"ContainerDied","Data":"5633f6c4153856f866f0d587cbf456f306c79dfa632c6ba5679541088760fd2d"} Oct 30 20:03:49.124430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.124362 4990 scope.go:122] "RemoveContainer" containerID="b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6" Oct 30 20:03:49.136127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.136109 4990 scope.go:122] "RemoveContainer" containerID="b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6" Oct 30 20:03:49.136383 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:03:49.136363 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6\": container with ID starting with b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6 not found: ID does not exist" containerID="b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6" Oct 30 20:03:49.136426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.136392 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6"} err="failed to get container status \"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6\": rpc error: code = NotFound desc = could not find container \"b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6\": container with ID starting with b599395bf7b12f6ea021a5296e08f05d5b1a68bac13fcc68ee8db5c5bb79fdb6 not found: ID does not exist" Oct 30 20:03:49.175141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.175081 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 20:03:49.180068 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.180039 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2l4nm"] Oct 30 20:03:49.219849 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.219743 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.763711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.763663 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" containerID="cri-o://04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59" gracePeriod=25 Oct 30 20:03:49.798256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:49.798136 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" containerID="cri-o://67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434" gracePeriod=25 Oct 30 20:03:50.401741 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:50.401701 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6577b72-e951-4092-a434-648c4dec7b60" path="/var/lib/kubelet/pods/a6577b72-e951-4092-a434-648c4dec7b60/volumes" Oct 30 20:03:50.402376 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:50.402357 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" path="/var/lib/kubelet/pods/eba5e617-1e5c-47bc-94ed-0a9937f4c1da/volumes" Oct 30 20:03:50.527632 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:50.527592 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.046492 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:52.046454 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.433311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:52.433250 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.772937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:52.772807 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.998865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:52.998831 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.659044 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:53.658991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.875583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:53.875540 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" containerID="cri-o://449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c" gracePeriod=25 Oct 30 20:03:53.902755 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:53.902719 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" containerID="cri-o://3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088" gracePeriod=25 Oct 30 20:03:54.592202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:54.592166 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:54.592202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:54.592199 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:55.225631 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:55.225593 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:55.686298 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:55.686260 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:55.834848 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:55.834814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.344047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:56.344009 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.639917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:56.639834 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:56.640079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:56.639835 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:57.480428 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:57.480393 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.639449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:57.639412 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.681627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:57.681597 4990 scope.go:122] "RemoveContainer" containerID="2022dfd8430bd69feb86818e3fa440cf44a07017b4a130bf142842b6714534fc" Oct 30 20:03:57.691738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:57.691714 4990 scope.go:122] "RemoveContainer" containerID="de881739e67cf33e72e2d0d52b2a6e8ed9a454f8a16db94b025e243a44894b14" Oct 30 20:03:59.318480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:03:59.318423 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:00.864693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:00.864652 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.508874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:02.508836 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.334961 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.334915 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.488627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.488590 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.595646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.595553 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 20:04:03.596495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.596440 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" containerID="cri-o://aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f" gracePeriod=30 Oct 30 20:04:03.645419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.645377 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.797536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:03.797500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.138067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:04.138027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.295498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:04.295464 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.592466 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:04.592423 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:04.592657 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:04.592428 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:04.593926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:04.593899 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.085363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.085326 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.096626 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:05.096582 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:04:05.199522 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199482 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk"] Oct 30 20:04:05.199920 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199907 4990 cpu_manager.go:414] "containerMap: removing container" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199921 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199936 4990 cpu_manager.go:414] "containerMap: removing container" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199941 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199957 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199963 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199970 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.199981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199975 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199986 4990 cpu_manager.go:414] "containerMap: removing container" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199990 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.199996 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7d227471-b67a-4e3f-8225-f26fd36709a5" containerName="revoke-keys" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200001 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d227471-b67a-4e3f-8225-f26fd36709a5" containerName="revoke-keys" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200008 4990 cpu_manager.go:414] "containerMap: removing container" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200012 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200021 4990 cpu_manager.go:414] "containerMap: removing container" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200025 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200036 4990 cpu_manager.go:414] "containerMap: removing container" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200040 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200112 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba5e617-1e5c-47bc-94ed-0a9937f4c1da" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200120 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfeddfa-aaf2-48bf-86c8-44554170b811" containerName="payload-pre-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200126 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200134 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d153d4-e042-4b49-a64a-25cc4e542a7a" containerName="payload-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200142 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae0d1d6-1cb9-4f69-ac9a-f6b72cd831fa" containerName="payload-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200150 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6577b72-e951-4092-a434-648c4dec7b60" containerName="payload-pre-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200156 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb15296f-4e20-44d2-a97e-bff775dc046f" containerName="istio-proxy" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200162 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d56405-39e4-48bd-9623-bf65d5d279aa" containerName="payload-processing" Oct 30 20:04:05.200239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.200167 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d227471-b67a-4e3f-8225-f26fd36709a5" containerName="revoke-keys" Oct 30 20:04:05.205330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.205309 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.221835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.221809 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.221954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.221859 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.222057 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.222040 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5g9\" (UniqueName: \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.233346 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.233321 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk"] Oct 30 20:04:05.278654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.278621 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:05.278845 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.278799 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:04:05.323626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.323591 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.323809 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.323645 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.323809 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.323778 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5g9\" (UniqueName: \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.324339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.324310 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.326187 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.326158 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.340806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.340748 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5g9\" (UniqueName: \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\") pod \"maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.521113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.521060 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:05.819872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:05.819839 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk"] Oct 30 20:04:05.823503 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:04:05.823463 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118085aa_e3ac_4d82_b4eb_2fde3edb92e7.slice/crio-7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead WatchSource:0}: Error finding container 7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead: Status 404 returned error can't find the container with id 7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead Oct 30 20:04:06.062184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.062149 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.253481 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.253440 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" event={"ID":"118085aa-e3ac-4d82-b4eb-2fde3edb92e7","Type":"ContainerStarted","Data":"41ee65de0f76aeb6b6e9266c5200d80a2f6580a599e880da4649244bd668077c"} Oct 30 20:04:06.253903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.253502 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" event={"ID":"118085aa-e3ac-4d82-b4eb-2fde3edb92e7","Type":"ContainerStarted","Data":"7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead"} Oct 30 20:04:06.286465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.286417 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" podStartSLOduration=1.286402703 podStartE2EDuration="1.286402703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:06.285839 +0000 UTC m=+3553.997364366" watchObservedRunningTime="2026-10-30 20:04:06.286402703 +0000 UTC m=+3553.997928069" Oct 30 20:04:06.639817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.639782 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:06.639996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:06.639787 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:07.261244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:07.261210 4990 generic.go:356] "Generic (PLEG): container finished" podID="118085aa-e3ac-4d82-b4eb-2fde3edb92e7" containerID="41ee65de0f76aeb6b6e9266c5200d80a2f6580a599e880da4649244bd668077c" exitCode=0 Oct 30 20:04:07.261667 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:07.261300 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" event={"ID":"118085aa-e3ac-4d82-b4eb-2fde3edb92e7","Type":"ContainerDied","Data":"41ee65de0f76aeb6b6e9266c5200d80a2f6580a599e880da4649244bd668077c"} Oct 30 20:04:07.588727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:07.588689 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.405449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.405420 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:08.453614 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.453586 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\" (UniqueName: \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\") pod \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " Oct 30 20:04:08.453780 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.453725 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\") pod \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " Oct 30 20:04:08.453827 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.453801 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\") pod \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\" (UID: \"118085aa-e3ac-4d82-b4eb-2fde3edb92e7\") " Oct 30 20:04:08.454207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.454173 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca" pod "118085aa-e3ac-4d82-b4eb-2fde3edb92e7" (UID: "118085aa-e3ac-4d82-b4eb-2fde3edb92e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:08.455771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.455738 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca" pod "118085aa-e3ac-4d82-b4eb-2fde3edb92e7" (UID: "118085aa-e3ac-4d82-b4eb-2fde3edb92e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:08.455891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.455806 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9" pod "118085aa-e3ac-4d82-b4eb-2fde3edb92e7" (UID: "118085aa-e3ac-4d82-b4eb-2fde3edb92e7"). InnerVolumeSpecName "kube-api-access-dw5g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:08.555513 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.555481 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.555513 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.555514 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5g9\" (UniqueName: \"kubernetes.io/projected/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-kube-api-access-dw5g9\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.555712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.555524 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118085aa-e3ac-4d82-b4eb-2fde3edb92e7-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.844253 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.844225 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 20:04:08.958725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958680 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958724 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958759 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958802 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958847 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958891 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958948 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.958993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.958980 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.959373 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959033 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\") pod \"5e283a85-0123-4942-9b49-84b639f8b5ee\" (UID: \"5e283a85-0123-4942-9b49-84b639f8b5ee\") " Oct 30 20:04:08.959456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959028 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.959516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959275 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.959516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959287 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.959516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959390 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.959688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.959611 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:08.961156 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.961121 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:08.961545 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.961514 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.961647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.961539 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:08.961647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:08.961591 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl" pod "5e283a85-0123-4942-9b49-84b639f8b5ee" (UID: "5e283a85-0123-4942-9b49-84b639f8b5ee"). InnerVolumeSpecName "kube-api-access-tsxtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:09.060656 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060616 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxtl\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-kube-api-access-tsxtl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060656 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060649 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060656 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060660 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e283a85-0123-4942-9b49-84b639f8b5ee-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060669 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060677 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060685 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060693 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060712 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e283a85-0123-4942-9b49-84b639f8b5ee-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.060910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.060732 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e283a85-0123-4942-9b49-84b639f8b5ee-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.064693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.064663 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 20:04:09.074603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.074572 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 20:04:09.203179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.203075 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.276275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.276248 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" Oct 30 20:04:09.276474 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.276249 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk" event={"ID":"118085aa-e3ac-4d82-b4eb-2fde3edb92e7","Type":"ContainerDied","Data":"7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead"} Oct 30 20:04:09.276474 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.276436 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a64ff1f1cf0192ae435d1d52ed802a030114a1328a9b4c175fa636b2da06ead" Oct 30 20:04:09.277806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.277783 4990 generic.go:356] "Generic (PLEG): container finished" podID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerID="aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f" exitCode=0 Oct 30 20:04:09.277947 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.277830 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" Oct 30 20:04:09.277947 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.277866 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" event={"ID":"5e283a85-0123-4942-9b49-84b639f8b5ee","Type":"ContainerDied","Data":"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f"} Oct 30 20:04:09.277947 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.277917 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx" event={"ID":"5e283a85-0123-4942-9b49-84b639f8b5ee","Type":"ContainerDied","Data":"c35ef7ef50c01718e89b7dd8d0811a3de413be397d27d4afa5d4074865fb8ebf"} Oct 30 20:04:09.278074 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.277947 4990 scope.go:122] "RemoveContainer" containerID="aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f" Oct 30 20:04:09.291384 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.291317 4990 scope.go:122] "RemoveContainer" containerID="aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f" Oct 30 20:04:09.291656 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:09.291638 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f\": container with ID starting with aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f not found: ID does not exist" containerID="aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f" Oct 30 20:04:09.291722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.291665 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f"} err="failed to get container status \"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f\": rpc error: code = NotFound desc = could not find container \"aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f\": container with ID starting with aa53426a6f1c952310c80616d12e168b5a060160c8e5857ac4e8e555fd93144f not found: ID does not exist" Oct 30 20:04:09.325518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.325487 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 20:04:09.329464 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.329433 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-kfhsx"] Oct 30 20:04:09.347004 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.346971 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.995939 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:09.995906 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 20:04:10.011149 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.011116 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 20:04:10.033319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.021397 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 20:04:10.081343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.081318 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 20:04:10.084788 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.084767 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 20:04:10.167284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.167248 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.174208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174185 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\" (UniqueName: \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\") pod \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " Oct 30 20:04:10.174328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174257 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\" (UniqueName: \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\") pod \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " Oct 30 20:04:10.174409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174330 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\") pod \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\" (UID: \"5485d987-e7d8-4ea8-9d66-0cd034a1c538\") " Oct 30 20:04:10.174409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174361 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\") pod \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\" (UID: \"dbe0109d-53ae-43d2-856d-8ded2e81f4cf\") " Oct 30 20:04:10.174782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174753 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume" pod "5485d987-e7d8-4ea8-9d66-0cd034a1c538" (UID: "5485d987-e7d8-4ea8-9d66-0cd034a1c538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:10.174913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.174800 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume" pod "dbe0109d-53ae-43d2-856d-8ded2e81f4cf" (UID: "dbe0109d-53ae-43d2-856d-8ded2e81f4cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:10.176534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.176511 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7" pod "5485d987-e7d8-4ea8-9d66-0cd034a1c538" (UID: "5485d987-e7d8-4ea8-9d66-0cd034a1c538"). InnerVolumeSpecName "kube-api-access-58ks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:10.176890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.176863 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf" pod "dbe0109d-53ae-43d2-856d-8ded2e81f4cf" (UID: "dbe0109d-53ae-43d2-856d-8ded2e81f4cf"). InnerVolumeSpecName "kube-api-access-nxrvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:10.275718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.275679 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxrvf\" (UniqueName: \"kubernetes.io/projected/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-kube-api-access-nxrvf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.275718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.275710 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5485d987-e7d8-4ea8-9d66-0cd034a1c538-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.275718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.275719 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe0109d-53ae-43d2-856d-8ded2e81f4cf-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.275718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.275727 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58ks7\" (UniqueName: \"kubernetes.io/projected/5485d987-e7d8-4ea8-9d66-0cd034a1c538-kube-api-access-58ks7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.286653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.286573 4990 generic.go:356] "Generic (PLEG): container finished" podID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerID="67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434" exitCode=0 Oct 30 20:04:10.286653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.286621 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" Oct 30 20:04:10.286850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.286663 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" event={"ID":"5485d987-e7d8-4ea8-9d66-0cd034a1c538","Type":"ContainerDied","Data":"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434"} Oct 30 20:04:10.286850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.286733 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq" event={"ID":"5485d987-e7d8-4ea8-9d66-0cd034a1c538","Type":"ContainerDied","Data":"aed9dd397597fb2def317e59510e62b3322534e2e36e359d0ba358ab16f806d9"} Oct 30 20:04:10.286850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.286764 4990 scope.go:122] "RemoveContainer" containerID="67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434" Oct 30 20:04:10.289608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.289529 4990 generic.go:356] "Generic (PLEG): container finished" podID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerID="04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59" exitCode=0 Oct 30 20:04:10.289608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.289564 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" event={"ID":"dbe0109d-53ae-43d2-856d-8ded2e81f4cf","Type":"ContainerDied","Data":"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59"} Oct 30 20:04:10.289821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.289624 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" event={"ID":"dbe0109d-53ae-43d2-856d-8ded2e81f4cf","Type":"ContainerDied","Data":"3e7fc2368801f65b86e7af583a0bb8439aa1006333b9b8f2194d3d5711ed969b"} Oct 30 20:04:10.289821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.289582 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5" Oct 30 20:04:10.302844 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.302672 4990 scope.go:122] "RemoveContainer" containerID="67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434" Oct 30 20:04:10.303259 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:10.303239 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434\": container with ID starting with 67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434 not found: ID does not exist" containerID="67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434" Oct 30 20:04:10.303329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.303265 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434"} err="failed to get container status \"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434\": rpc error: code = NotFound desc = could not find container \"67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434\": container with ID starting with 67c3ae8c3ecd24dcf52f4e6558117cbe2490c3f98fcaf474ef6ee1f067a6b434 not found: ID does not exist" Oct 30 20:04:10.303329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.303281 4990 scope.go:122] "RemoveContainer" containerID="04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59" Oct 30 20:04:10.312566 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.312550 4990 scope.go:122] "RemoveContainer" containerID="04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59" Oct 30 20:04:10.312882 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:10.312862 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59\": container with ID starting with 04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59 not found: ID does not exist" containerID="04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59" Oct 30 20:04:10.312945 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.312889 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59"} err="failed to get container status \"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59\": rpc error: code = NotFound desc = could not find container \"04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59\": container with ID starting with 04fc31952c9b9d95c2357c982b84d3ee56fcffb60eaac49c77178308c7264f59 not found: ID does not exist" Oct 30 20:04:10.348466 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.348439 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 20:04:10.351765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.351742 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-870193-7b56c9cdd7-n97pq"] Oct 30 20:04:10.383517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.383491 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 20:04:10.389993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.389972 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-870193-6c847f86c8-tvcc5"] Oct 30 20:04:10.399387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.399357 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" path="/var/lib/kubelet/pods/5485d987-e7d8-4ea8-9d66-0cd034a1c538/volumes" Oct 30 20:04:10.399873 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.399851 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" path="/var/lib/kubelet/pods/5e283a85-0123-4942-9b49-84b639f8b5ee/volumes" Oct 30 20:04:10.400320 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:10.400301 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" path="/var/lib/kubelet/pods/dbe0109d-53ae-43d2-856d-8ded2e81f4cf/volumes" Oct 30 20:04:11.016417 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:11.016376 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.563680 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:12.563643 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.085902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.085865 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.148852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.148784 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 20:04:14.176133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.176110 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 20:04:14.215873 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.215841 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\" (UniqueName: \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\") pod \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " Oct 30 20:04:14.216054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.215948 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\") pod \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\" (UID: \"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803\") " Oct 30 20:04:14.216325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.216301 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume" pod "a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" (UID: "a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:14.218351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.218316 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24" pod "a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" (UID: "a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803"). InnerVolumeSpecName "kube-api-access-7gq24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:14.316579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.316545 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\" (UniqueName: \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\") pod \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " Oct 30 20:04:14.316756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.316638 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\") pod \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\" (UID: \"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee\") " Oct 30 20:04:14.317026 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.317006 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gq24\" (UniqueName: \"kubernetes.io/projected/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-kube-api-access-7gq24\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:14.317125 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.317032 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:14.317125 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.317047 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume" pod "f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" (UID: "f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:14.318829 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.318799 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v" pod "f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" (UID: "f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee"). InnerVolumeSpecName "kube-api-access-b5r6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:14.319501 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.319478 4990 generic.go:356] "Generic (PLEG): container finished" podID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerID="449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c" exitCode=0 Oct 30 20:04:14.319597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.319525 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" Oct 30 20:04:14.319597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.319556 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" event={"ID":"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803","Type":"ContainerDied","Data":"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c"} Oct 30 20:04:14.319701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.319612 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2" event={"ID":"a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803","Type":"ContainerDied","Data":"61edfa2ccc109659c05dcffa22dd0e0eaa3ddb1f214a13d46cb0e0892624b5ad"} Oct 30 20:04:14.319701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.319640 4990 scope.go:122] "RemoveContainer" containerID="449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c" Oct 30 20:04:14.321048 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.321026 4990 generic.go:356] "Generic (PLEG): container finished" podID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerID="3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088" exitCode=0 Oct 30 20:04:14.321175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.321073 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" Oct 30 20:04:14.321175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.321113 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" event={"ID":"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee","Type":"ContainerDied","Data":"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088"} Oct 30 20:04:14.321175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.321155 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh" event={"ID":"f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee","Type":"ContainerDied","Data":"5741e6c96f84d70235caf610155426b04d876f3aa1e37b94002e6352b2a74873"} Oct 30 20:04:14.329672 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.329649 4990 scope.go:122] "RemoveContainer" containerID="449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c" Oct 30 20:04:14.329983 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:14.329960 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c\": container with ID starting with 449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c not found: ID does not exist" containerID="449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c" Oct 30 20:04:14.330067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.329990 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c"} err="failed to get container status \"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c\": rpc error: code = NotFound desc = could not find container \"449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c\": container with ID starting with 449f2a7ca3e1cd445914f19035642593f64081b3e97dd1572df3cd2a974f5e3c not found: ID does not exist" Oct 30 20:04:14.330067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.330011 4990 scope.go:122] "RemoveContainer" containerID="3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088" Oct 30 20:04:14.339123 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.339107 4990 scope.go:122] "RemoveContainer" containerID="3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088" Oct 30 20:04:14.339376 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:14.339353 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088\": container with ID starting with 3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088 not found: ID does not exist" containerID="3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088" Oct 30 20:04:14.339455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.339386 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088"} err="failed to get container status \"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088\": rpc error: code = NotFound desc = could not find container \"3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088\": container with ID starting with 3a9c3fd470ab3dc6086b31357a2333b5951bcf3a66fce32335f1fe8f78f63088 not found: ID does not exist" Oct 30 20:04:14.375063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.375027 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 20:04:14.379041 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.379016 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5a0a17-66c4d7f7c8-rdpp2"] Oct 30 20:04:14.399133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.399056 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" path="/var/lib/kubelet/pods/a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803/volumes" Oct 30 20:04:14.407434 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.407411 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 20:04:14.411136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.411116 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5a0a17-6665786cc9-j52vh"] Oct 30 20:04:14.418232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.418213 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5r6v\" (UniqueName: \"kubernetes.io/projected/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-kube-api-access-b5r6v\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:14.418302 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.418234 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:14.820792 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.820759 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 20:04:14.970619 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:14.970581 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.112872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.112764 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.532578 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532502 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4"] Oct 30 20:04:15.532890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532875 4990 cpu_manager.go:414] "containerMap: removing container" podUID="118085aa-e3ac-4d82-b4eb-2fde3edb92e7" containerName="revoke-keys" Oct 30 20:04:15.532932 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532891 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118085aa-e3ac-4d82-b4eb-2fde3edb92e7" containerName="revoke-keys" Oct 30 20:04:15.532932 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532930 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532935 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532951 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532956 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532964 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532969 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532977 4990 cpu_manager.go:414] "containerMap: removing container" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532982 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532991 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" Oct 30 20:04:15.533005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.532995 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533047 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c8c57a-da7b-4b6d-91ec-a2f2d1ed8803" containerName="payload-processing" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533059 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="118085aa-e3ac-4d82-b4eb-2fde3edb92e7" containerName="revoke-keys" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533066 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5485d987-e7d8-4ea8-9d66-0cd034a1c538" containerName="payload-pre-processing" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533074 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e283a85-0123-4942-9b49-84b639f8b5ee" containerName="istio-proxy" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533079 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" containerName="payload-pre-processing" Oct 30 20:04:15.533325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.533083 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe0109d-53ae-43d2-856d-8ded2e81f4cf" containerName="payload-processing" Oct 30 20:04:15.537470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.537454 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.556429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.556400 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4"] Oct 30 20:04:15.608409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.608375 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 20:04:15.630382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.630351 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt58h\" (UniqueName: \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.630554 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.630403 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.630608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.630587 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.637310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.637289 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:15.637397 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.637362 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:04:15.731837 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.731805 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.732012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.731852 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt58h\" (UniqueName: \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.732012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.731884 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.732479 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.732456 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.734276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.734248 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.748361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.748336 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt58h\" (UniqueName: \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\") pod \"maas-api-revoke-sub-b180a7e9-cpps4\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.758803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.758766 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.850778 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.850750 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:15.908861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:15.908822 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:16.059052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:16.059020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:16.159609 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:16.159530 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4"] Oct 30 20:04:16.163460 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:04:16.163429 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a14c8c3_d0c2_48f0_9f9d_0e7cef498d5a.slice/crio-b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423 WatchSource:0}: Error finding container b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423: Status 404 returned error can't find the container with id b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423 Oct 30 20:04:16.338162 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:16.338121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" event={"ID":"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a","Type":"ContainerStarted","Data":"b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423"} Oct 30 20:04:16.398908 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:16.398875 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee" path="/var/lib/kubelet/pods/f49b6b2e-cb5e-4d0c-b443-2f7daa5189ee/volumes" Oct 30 20:04:17.347123 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:17.347063 4990 generic.go:356] "Generic (PLEG): container finished" podID="9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" containerID="8b3f1b9492c5963affaff3a806666b70274d157ee0f2e261bedf89efc8b851b3" exitCode=0 Oct 30 20:04:17.347686 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:17.347129 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" event={"ID":"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a","Type":"ContainerDied","Data":"8b3f1b9492c5963affaff3a806666b70274d157ee0f2e261bedf89efc8b851b3"} Oct 30 20:04:18.476171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.476142 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:18.557073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.557042 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\") pod \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " Oct 30 20:04:18.557244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.557141 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\" (UniqueName: \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\") pod \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " Oct 30 20:04:18.557244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.557188 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\") pod \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\" (UID: \"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a\") " Oct 30 20:04:18.557592 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.557564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca" pod "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" (UID: "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:18.559259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.559230 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca" pod "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" (UID: "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:18.559349 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.559264 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h" pod "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" (UID: "9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a"). InnerVolumeSpecName "kube-api-access-bt58h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:18.658472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.658393 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:18.658472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.658423 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:18.658472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:18.658441 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt58h\" (UniqueName: \"kubernetes.io/projected/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a-kube-api-access-bt58h\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.361605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:19.361566 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" Oct 30 20:04:19.361788 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:19.361601 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4" event={"ID":"9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a","Type":"ContainerDied","Data":"b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423"} Oct 30 20:04:19.361788 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:19.361651 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b07417a7ce72c6831066d9bc0ae587167fadb4a1e5ff4c0c2443b5cf965da423" Oct 30 20:04:19.821144 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:19.821082 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" containerID="cri-o://d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566" gracePeriod=25 Oct 30 20:04:20.609356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:20.609312 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" containerID="cri-o://057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c" gracePeriod=25 Oct 30 20:04:22.641659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:22.641610 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:24.085347 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.085298 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" containerID="cri-o://957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c" gracePeriod=45 Oct 30 20:04:24.098523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.098486 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" containerID="cri-o://0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0" gracePeriod=45 Oct 30 20:04:24.121256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.121218 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused" Oct 30 20:04:24.351227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.351168 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 20:04:24.368542 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.368520 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 20:04:24.396616 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.396588 4990 generic.go:356] "Generic (PLEG): container finished" podID="f1bd696f-6728-43df-af50-c77f9efb9986" containerID="0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0" exitCode=0 Oct 30 20:04:24.396856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.396837 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" Oct 30 20:04:24.398753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.398728 4990 generic.go:356] "Generic (PLEG): container finished" podID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerID="957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c" exitCode=0 Oct 30 20:04:24.398854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.398828 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" Oct 30 20:04:24.403721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.403693 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" event={"ID":"f1bd696f-6728-43df-af50-c77f9efb9986","Type":"ContainerDied","Data":"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0"} Oct 30 20:04:24.403846 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.403737 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg" event={"ID":"f1bd696f-6728-43df-af50-c77f9efb9986","Type":"ContainerDied","Data":"e00a8cb22d5f1d4d0e11d19f33fc098936b506f679a7bbf03101b287a8114b88"} Oct 30 20:04:24.403846 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.403750 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" event={"ID":"5222eace-2251-497e-8a42-8b506a6ac4a2","Type":"ContainerDied","Data":"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c"} Oct 30 20:04:24.403846 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.403764 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d" event={"ID":"5222eace-2251-497e-8a42-8b506a6ac4a2","Type":"ContainerDied","Data":"ea4afb0710c5e1bef635301d55f3af2af8d634f7345f6d51b9db790c17cf6a5e"} Oct 30 20:04:24.403846 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.403794 4990 scope.go:122] "RemoveContainer" containerID="0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0" Oct 30 20:04:24.414018 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.413994 4990 scope.go:122] "RemoveContainer" containerID="0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0" Oct 30 20:04:24.414374 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:24.414353 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0\": container with ID starting with 0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0 not found: ID does not exist" containerID="0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0" Oct 30 20:04:24.414487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.414383 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0"} err="failed to get container status \"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0\": rpc error: code = NotFound desc = could not find container \"0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0\": container with ID starting with 0a4360a388f7b82088387fda4914cb572ceffd7917bbaaf66d77aacbddb1eff0 not found: ID does not exist" Oct 30 20:04:24.414487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.414401 4990 scope.go:122] "RemoveContainer" containerID="957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c" Oct 30 20:04:24.418832 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.418808 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.418953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.418851 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.418953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.418898 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\" (UniqueName: \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.418953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.418930 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.419283 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.418962 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.419283 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419002 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\") pod \"5222eace-2251-497e-8a42-8b506a6ac4a2\" (UID: \"5222eace-2251-497e-8a42-8b506a6ac4a2\") " Oct 30 20:04:24.419418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419366 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.419418 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419383 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.419540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419489 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.419598 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419537 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.419598 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.419566 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.421637 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.421601 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "kube-api-access-6ggr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:24.421761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.421683 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.421861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.421843 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs" pod "5222eace-2251-497e-8a42-8b506a6ac4a2" (UID: "5222eace-2251-497e-8a42-8b506a6ac4a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:24.425659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.425469 4990 scope.go:122] "RemoveContainer" containerID="957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c" Oct 30 20:04:24.425847 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:24.425819 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c\": container with ID starting with 957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c not found: ID does not exist" containerID="957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c" Oct 30 20:04:24.426410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.425859 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c"} err="failed to get container status \"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c\": rpc error: code = NotFound desc = could not find container \"957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c\": container with ID starting with 957213aa2260365751010749a87a27c1702be9d6145d5a2bdfca0ecb014f698c not found: ID does not exist" Oct 30 20:04:24.520330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520299 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520369 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520406 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\" (UniqueName: \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520440 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520493 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520537 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\") pod \"f1bd696f-6728-43df-af50-c77f9efb9986\" (UID: \"f1bd696f-6728-43df-af50-c77f9efb9986\") " Oct 30 20:04:24.520870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520568 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.520870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520767 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.520870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520855 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520943 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520957 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520967 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520978 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.520991 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggr7\" (UniqueName: \"kubernetes.io/projected/5222eace-2251-497e-8a42-8b506a6ac4a2-kube-api-access-6ggr7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.521001 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5222eace-2251-497e-8a42-8b506a6ac4a2-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.521112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.521010 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5222eace-2251-497e-8a42-8b506a6ac4a2-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.522615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.522589 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.522726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.522633 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:24.522726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.522670 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99" pod "f1bd696f-6728-43df-af50-c77f9efb9986" (UID: "f1bd696f-6728-43df-af50-c77f9efb9986"). InnerVolumeSpecName "kube-api-access-w7m99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:24.621754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.621680 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7m99\" (UniqueName: \"kubernetes.io/projected/f1bd696f-6728-43df-af50-c77f9efb9986-kube-api-access-w7m99\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.621754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.621708 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bd696f-6728-43df-af50-c77f9efb9986-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.621754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.621720 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bd696f-6728-43df-af50-c77f9efb9986-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.739262 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.739227 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 20:04:24.748933 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.748900 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/simulator-w5-e67b84-kserve-7966485c8b-8mtlg"] Oct 30 20:04:24.779439 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.779411 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 20:04:24.783504 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:24.783478 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/premium-w5-e67b84-kserve-947b46d59-ks87d"] Oct 30 20:04:25.022396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.022298 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" containerID="cri-o://743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e" gracePeriod=45 Oct 30 20:04:25.035713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.035682 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" containerID="cri-o://e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad" gracePeriod=45 Oct 30 20:04:25.047963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.047930 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" containerID="cri-o://3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea" gracePeriod=45 Oct 30 20:04:25.304660 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.304634 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 20:04:25.335049 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.335025 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 20:04:25.337862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.337842 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 20:04:25.410677 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.410643 4990 generic.go:356] "Generic (PLEG): container finished" podID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerID="743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e" exitCode=0 Oct 30 20:04:25.410881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.410795 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" Oct 30 20:04:25.410947 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.410929 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" event={"ID":"0746b3fc-6eba-4f5f-bfee-2dde967613f6","Type":"ContainerDied","Data":"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e"} Oct 30 20:04:25.411005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.410964 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t" event={"ID":"0746b3fc-6eba-4f5f-bfee-2dde967613f6","Type":"ContainerDied","Data":"432350c3ae033e01a08e682bae48b2a5c91d5e9f37b3e662a5b3da67b30ce74e"} Oct 30 20:04:25.411005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.410989 4990 scope.go:122] "RemoveContainer" containerID="743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e" Oct 30 20:04:25.412528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.412506 4990 generic.go:356] "Generic (PLEG): container finished" podID="31e95e0b-6c18-42af-a847-a25520a997ec" containerID="3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea" exitCode=0 Oct 30 20:04:25.412679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.412571 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" Oct 30 20:04:25.412679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.412598 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" event={"ID":"31e95e0b-6c18-42af-a847-a25520a997ec","Type":"ContainerDied","Data":"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea"} Oct 30 20:04:25.412679 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.412644 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn" event={"ID":"31e95e0b-6c18-42af-a847-a25520a997ec","Type":"ContainerDied","Data":"67fbf7f93ee6b1dc8ee4ced85ba75ab698c733e318b3911dffe5e4f63fbc509a"} Oct 30 20:04:25.415590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.415563 4990 generic.go:356] "Generic (PLEG): container finished" podID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerID="e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad" exitCode=0 Oct 30 20:04:25.415728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.415625 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" Oct 30 20:04:25.415728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.415637 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" event={"ID":"a4e06bd4-e6fe-477c-8385-64ba396f02c8","Type":"ContainerDied","Data":"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad"} Oct 30 20:04:25.415728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.415665 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg" event={"ID":"a4e06bd4-e6fe-477c-8385-64ba396f02c8","Type":"ContainerDied","Data":"bb9e6e451ca3dc1d9349f7e76001d43271f53d37ac031942ed82344f70f59bdf"} Oct 30 20:04:25.429766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429719 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\" (UniqueName: \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.429962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429794 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.429962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429828 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.429962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429870 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.429962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429898 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.429975 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\" (UniqueName: \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430009 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430060 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430105 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430154 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\" (UniqueName: \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430171 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.430232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430184 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430268 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430306 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430351 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430397 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\") pod \"31e95e0b-6c18-42af-a847-a25520a997ec\" (UID: \"31e95e0b-6c18-42af-a847-a25520a997ec\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430436 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\") pod \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\" (UID: \"0746b3fc-6eba-4f5f-bfee-2dde967613f6\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430465 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430508 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.430595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430519 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\") pod \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\" (UID: \"a4e06bd4-e6fe-477c-8385-64ba396f02c8\") " Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430701 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430706 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430900 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430926 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430948 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430963 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.430988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.430978 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.431430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.431053 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.431430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.431107 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.431430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.431409 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.431758 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.431683 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.432305 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.432203 4990 scope.go:122] "RemoveContainer" containerID="743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e" Oct 30 20:04:25.432966 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:25.432682 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e\": container with ID starting with 743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e not found: ID does not exist" containerID="743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e" Oct 30 20:04:25.432966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.432721 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e"} err="failed to get container status \"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e\": rpc error: code = NotFound desc = could not find container \"743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e\": container with ID starting with 743ab9c7455e00f23086531da65e5f3db7d1b8fe086c2d5a664598371468ea4e not found: ID does not exist" Oct 30 20:04:25.432966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.432748 4990 scope.go:122] "RemoveContainer" containerID="3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea" Oct 30 20:04:25.434382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.434045 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "kube-api-access-mnr66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:25.434382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.434061 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.434382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.434188 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:25.435421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.435397 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.435421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.435409 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs" pod "0746b3fc-6eba-4f5f-bfee-2dde967613f6" (UID: "0746b3fc-6eba-4f5f-bfee-2dde967613f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:25.435568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.435471 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:25.435568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.435502 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:25.436898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.436855 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8" pod "a4e06bd4-e6fe-477c-8385-64ba396f02c8" (UID: "a4e06bd4-e6fe-477c-8385-64ba396f02c8"). InnerVolumeSpecName "kube-api-access-nlzg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:25.437294 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.437266 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv" pod "31e95e0b-6c18-42af-a847-a25520a997ec" (UID: "31e95e0b-6c18-42af-a847-a25520a997ec"). InnerVolumeSpecName "kube-api-access-vxmzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:25.446835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.446810 4990 scope.go:122] "RemoveContainer" containerID="3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea" Oct 30 20:04:25.447179 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:25.447154 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea\": container with ID starting with 3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea not found: ID does not exist" containerID="3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea" Oct 30 20:04:25.447287 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.447192 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea"} err="failed to get container status \"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea\": rpc error: code = NotFound desc = could not find container \"3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea\": container with ID starting with 3836f1e1f86e3af1ed1571fc98dc453149551d465dd27586fffeb6e53f68cbea not found: ID does not exist" Oct 30 20:04:25.447287 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.447216 4990 scope.go:122] "RemoveContainer" containerID="e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad" Oct 30 20:04:25.456577 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.456558 4990 scope.go:122] "RemoveContainer" containerID="e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad" Oct 30 20:04:25.456843 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:25.456825 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad\": container with ID starting with e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad not found: ID does not exist" containerID="e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad" Oct 30 20:04:25.456897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.456848 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad"} err="failed to get container status \"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad\": rpc error: code = NotFound desc = could not find container \"e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad\": container with ID starting with e71a23169863e33084eb2b4a3af7c30b7181d713517095233f75737501d6e7ad not found: ID does not exist" Oct 30 20:04:25.531678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531643 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531672 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531682 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnr66\" (UniqueName: \"kubernetes.io/projected/0746b3fc-6eba-4f5f-bfee-2dde967613f6-kube-api-access-mnr66\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531691 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e95e0b-6c18-42af-a847-a25520a997ec-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531700 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531709 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531716 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0746b3fc-6eba-4f5f-bfee-2dde967613f6-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531725 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531732 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531739 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e06bd4-e6fe-477c-8385-64ba396f02c8-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531747 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzg8\" (UniqueName: \"kubernetes.io/projected/a4e06bd4-e6fe-477c-8385-64ba396f02c8-kube-api-access-nlzg8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531756 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e95e0b-6c18-42af-a847-a25520a997ec-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531764 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmzv\" (UniqueName: \"kubernetes.io/projected/31e95e0b-6c18-42af-a847-a25520a997ec-kube-api-access-vxmzv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.531918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.531773 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0746b3fc-6eba-4f5f-bfee-2dde967613f6-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:25.757234 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.757199 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 20:04:25.766576 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.766546 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-2-w4-b97379-kserve-5cf66d898f-dgq6t"] Oct 30 20:04:25.790527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.790498 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 20:04:25.796189 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.796162 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/simulator-w4-b97379-kserve-74794cfbf8-4qqpn"] Oct 30 20:04:25.824712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.824687 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 20:04:25.829153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:25.829130 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/premium-w4-b97379-kserve-88dd7cdc9-g4lqg"] Oct 30 20:04:26.398891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.398805 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" path="/var/lib/kubelet/pods/0746b3fc-6eba-4f5f-bfee-2dde967613f6/volumes" Oct 30 20:04:26.399467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.399444 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" path="/var/lib/kubelet/pods/31e95e0b-6c18-42af-a847-a25520a997ec/volumes" Oct 30 20:04:26.399894 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.399875 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" path="/var/lib/kubelet/pods/5222eace-2251-497e-8a42-8b506a6ac4a2/volumes" Oct 30 20:04:26.400621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.400599 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" path="/var/lib/kubelet/pods/a4e06bd4-e6fe-477c-8385-64ba396f02c8/volumes" Oct 30 20:04:26.401054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.401035 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" path="/var/lib/kubelet/pods/f1bd696f-6728-43df-af50-c77f9efb9986/volumes" Oct 30 20:04:26.716282 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:26.716193 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:28.860620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:28.860564 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.007138 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:29.007082 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.653038 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:29.652999 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.819354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:29.819316 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.619460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.619424 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 20:04:30.619970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.619619 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" podUID="cb134f79-e7dc-42aa-9960-0e2843766d5a" containerName="maas-api" containerID="cri-o://670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94" gracePeriod=30 Oct 30 20:04:30.737975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.737942 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.874997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.874935 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 20:04:30.989333 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.989299 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\") pod \"cb134f79-e7dc-42aa-9960-0e2843766d5a\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " Oct 30 20:04:30.989505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.989380 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\" (UniqueName: \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\") pod \"cb134f79-e7dc-42aa-9960-0e2843766d5a\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " Oct 30 20:04:30.989505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.989405 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\") pod \"cb134f79-e7dc-42aa-9960-0e2843766d5a\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " Oct 30 20:04:30.989505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.989498 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") pod \"cb134f79-e7dc-42aa-9960-0e2843766d5a\" (UID: \"cb134f79-e7dc-42aa-9960-0e2843766d5a\") " Oct 30 20:04:30.989833 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.989806 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca" pod "cb134f79-e7dc-42aa-9960-0e2843766d5a" (UID: "cb134f79-e7dc-42aa-9960-0e2843766d5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:30.991678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.991653 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7" pod "cb134f79-e7dc-42aa-9960-0e2843766d5a" (UID: "cb134f79-e7dc-42aa-9960-0e2843766d5a"). InnerVolumeSpecName "kube-api-access-589c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:30.991961 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.991944 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls" pod "cb134f79-e7dc-42aa-9960-0e2843766d5a" (UID: "cb134f79-e7dc-42aa-9960-0e2843766d5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:30.992039 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:30.991967 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs" pod "cb134f79-e7dc-42aa-9960-0e2843766d5a" (UID: "cb134f79-e7dc-42aa-9960-0e2843766d5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:31.090368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.090328 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589c7\" (UniqueName: \"kubernetes.io/projected/cb134f79-e7dc-42aa-9960-0e2843766d5a-kube-api-access-589c7\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:31.090368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.090361 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb134f79-e7dc-42aa-9960-0e2843766d5a-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:31.090368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.090371 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:31.090368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.090380 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb134f79-e7dc-42aa-9960-0e2843766d5a-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:31.237887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.237800 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.304668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.304635 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk"] Oct 30 20:04:31.307346 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.307321 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b97379-xcghk"] Oct 30 20:04:31.456683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.456651 4990 generic.go:356] "Generic (PLEG): container finished" podID="cb134f79-e7dc-42aa-9960-0e2843766d5a" containerID="670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94" exitCode=0 Oct 30 20:04:31.456872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.456694 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" Oct 30 20:04:31.456872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.456729 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" event={"ID":"cb134f79-e7dc-42aa-9960-0e2843766d5a","Type":"ContainerDied","Data":"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94"} Oct 30 20:04:31.456872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.456779 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz" event={"ID":"cb134f79-e7dc-42aa-9960-0e2843766d5a","Type":"ContainerDied","Data":"3d1b05b906c209e1d6b7be737d7308e2902713fd78e9c6b7aae5f8871f56713a"} Oct 30 20:04:31.456872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.456802 4990 scope.go:122] "RemoveContainer" containerID="670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94" Oct 30 20:04:31.467229 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.467206 4990 scope.go:122] "RemoveContainer" containerID="670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94" Oct 30 20:04:31.467521 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:31.467502 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94\": container with ID starting with 670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94 not found: ID does not exist" containerID="670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94" Oct 30 20:04:31.467590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.467532 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94"} err="failed to get container status \"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94\": rpc error: code = NotFound desc = could not find container \"670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94\": container with ID starting with 670ccf143c27fda2f7954a086d44a70b5415ea7df80a9b51cd257b229ef39f94 not found: ID does not exist" Oct 30 20:04:31.509135 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.509032 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.525661 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.525627 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 20:04:31.530678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.530654 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b97379-59c4cd895-cdxgz"] Oct 30 20:04:31.805104 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.805061 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.953615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:31.953574 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.096866 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:32.096776 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.401375 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:32.401289 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118085aa-e3ac-4d82-b4eb-2fde3edb92e7" path="/var/lib/kubelet/pods/118085aa-e3ac-4d82-b4eb-2fde3edb92e7/volumes" Oct 30 20:04:32.401694 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:32.401679 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb134f79-e7dc-42aa-9960-0e2843766d5a" path="/var/lib/kubelet/pods/cb134f79-e7dc-42aa-9960-0e2843766d5a/volumes" Oct 30 20:04:32.642165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:32.642125 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:33.730238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:33.730186 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.196108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:34.196058 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.344757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:34.344722 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.789977 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:34.789943 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.095351 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:35.095309 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:04:36.716627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:36.716587 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:04:38.839148 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:38.839108 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.001415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:39.001376 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.163368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:39.163235 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.338551 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:39.338514 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.576956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:39.576923 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.071222 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.071197 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 20:04:40.164335 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.164242 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\") pod \"c79fb900-6199-42f1-83c2-3dcdb1671412\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " Oct 30 20:04:40.164335 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.164292 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\" (UniqueName: \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\") pod \"c79fb900-6199-42f1-83c2-3dcdb1671412\" (UID: \"c79fb900-6199-42f1-83c2-3dcdb1671412\") " Oct 30 20:04:40.164616 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.164591 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume" pod "c79fb900-6199-42f1-83c2-3dcdb1671412" (UID: "c79fb900-6199-42f1-83c2-3dcdb1671412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:40.166552 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.166526 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df" pod "c79fb900-6199-42f1-83c2-3dcdb1671412" (UID: "c79fb900-6199-42f1-83c2-3dcdb1671412"). InnerVolumeSpecName "kube-api-access-p48df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:40.264885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.264846 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79fb900-6199-42f1-83c2-3dcdb1671412-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:40.264885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.264880 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p48df\" (UniqueName: \"kubernetes.io/projected/c79fb900-6199-42f1-83c2-3dcdb1671412-kube-api-access-p48df\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:40.516752 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.516668 4990 generic.go:356] "Generic (PLEG): container finished" podID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerID="d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566" exitCode=0 Oct 30 20:04:40.516752 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.516715 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" Oct 30 20:04:40.516957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.516753 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" event={"ID":"c79fb900-6199-42f1-83c2-3dcdb1671412","Type":"ContainerDied","Data":"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566"} Oct 30 20:04:40.516957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.516804 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh" event={"ID":"c79fb900-6199-42f1-83c2-3dcdb1671412","Type":"ContainerDied","Data":"da266d0bfee9775e8a37ae7eb4919cdb4d3c0453fd4b5902f2bc26ac4e863478"} Oct 30 20:04:40.516957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.516841 4990 scope.go:122] "RemoveContainer" containerID="d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566" Oct 30 20:04:40.526438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.526419 4990 scope.go:122] "RemoveContainer" containerID="d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566" Oct 30 20:04:40.526741 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:40.526711 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566\": container with ID starting with d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566 not found: ID does not exist" containerID="d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566" Oct 30 20:04:40.526850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.526745 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566"} err="failed to get container status \"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566\": rpc error: code = NotFound desc = could not find container \"d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566\": container with ID starting with d266848f72fa48837d395c6ab1e3f3dbf2d8eff5193246d94ee57d49168a2566 not found: ID does not exist" Oct 30 20:04:40.564935 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.564902 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 20:04:40.568185 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.568161 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e67b84-684f75b9ff-nwjkh"] Oct 30 20:04:40.859739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.859714 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 20:04:40.870216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.870190 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\") pod \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " Oct 30 20:04:40.870365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.870236 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\" (UniqueName: \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\") pod \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\" (UID: \"f6dbe448-a748-4264-ba38-8ed7dbc12e4d\") " Oct 30 20:04:40.870555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.870532 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume" pod "f6dbe448-a748-4264-ba38-8ed7dbc12e4d" (UID: "f6dbe448-a748-4264-ba38-8ed7dbc12e4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:40.872458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.872425 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4" pod "f6dbe448-a748-4264-ba38-8ed7dbc12e4d" (UID: "f6dbe448-a748-4264-ba38-8ed7dbc12e4d"). InnerVolumeSpecName "kube-api-access-cdxv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:40.884741 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.884704 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.971728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.971692 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxv4\" (UniqueName: \"kubernetes.io/projected/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-kube-api-access-cdxv4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:40.971728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:40.971732 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6dbe448-a748-4264-ba38-8ed7dbc12e4d-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:04:41.041364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.041253 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.252680 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.252625 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.523902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.523867 4990 generic.go:356] "Generic (PLEG): container finished" podID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerID="057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c" exitCode=0 Oct 30 20:04:41.524117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.523914 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" Oct 30 20:04:41.524117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.523961 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" event={"ID":"f6dbe448-a748-4264-ba38-8ed7dbc12e4d","Type":"ContainerDied","Data":"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c"} Oct 30 20:04:41.524117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.524022 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z" event={"ID":"f6dbe448-a748-4264-ba38-8ed7dbc12e4d","Type":"ContainerDied","Data":"e9f0be305b148edc181ca8cd6de4214b0520f2f825c7b8bb99455d4268f9e278"} Oct 30 20:04:41.524117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.524050 4990 scope.go:122] "RemoveContainer" containerID="057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c" Oct 30 20:04:41.534325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.534294 4990 scope.go:122] "RemoveContainer" containerID="057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c" Oct 30 20:04:41.534577 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:04:41.534556 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c\": container with ID starting with 057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c not found: ID does not exist" containerID="057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c" Oct 30 20:04:41.534627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.534586 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c"} err="failed to get container status \"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c\": rpc error: code = NotFound desc = could not find container \"057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c\": container with ID starting with 057c18b555bd96b2ffbae9ca2f49ff2bcbb516167eae8bd69b924423c5de715c not found: ID does not exist" Oct 30 20:04:41.569047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.569018 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 20:04:41.578605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:41.578578 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b97379-95fdb8c-l2h7z"] Oct 30 20:04:42.399624 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:42.399582 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" path="/var/lib/kubelet/pods/c79fb900-6199-42f1-83c2-3dcdb1671412/volumes" Oct 30 20:04:42.400055 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:42.400036 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" path="/var/lib/kubelet/pods/f6dbe448-a748-4264-ba38-8ed7dbc12e4d/volumes" Oct 30 20:04:47.027394 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:47.027353 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9"] Oct 30 20:04:47.037365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:47.037336 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db458b49-fnsr9"] Oct 30 20:04:48.399554 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:48.399514 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbe93e8-fb40-4fea-bce0-fbefb143b0ff" path="/var/lib/kubelet/pods/ebbe93e8-fb40-4fea-bce0-fbefb143b0ff/volumes" Oct 30 20:04:54.703281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:54.703251 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:04:54.711678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:54.711654 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:04:54.743913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:54.743889 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:04:54.751911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:54.751892 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:04:55.558277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558243 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:04:55.558600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558587 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558603 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558616 4990 cpu_manager.go:414] "containerMap: removing container" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558621 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558631 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558636 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" Oct 30 20:04:55.558654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558653 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558659 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558665 4990 cpu_manager.go:414] "containerMap: removing container" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558669 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558679 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558684 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558702 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558709 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558726 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" containerName="revoke-keys" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558733 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" containerName="revoke-keys" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558744 4990 cpu_manager.go:414] "containerMap: removing container" podUID="cb134f79-e7dc-42aa-9960-0e2843766d5a" containerName="maas-api" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558748 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb134f79-e7dc-42aa-9960-0e2843766d5a" containerName="maas-api" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558811 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dbe448-a748-4264-ba38-8ed7dbc12e4d" containerName="payload-pre-processing" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558821 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" containerName="revoke-keys" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558830 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb134f79-e7dc-42aa-9960-0e2843766d5a" containerName="maas-api" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558837 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e06bd4-e6fe-477c-8385-64ba396f02c8" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558842 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="5222eace-2251-497e-8a42-8b506a6ac4a2" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558847 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0746b3fc-6eba-4f5f-bfee-2dde967613f6" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558854 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bd696f-6728-43df-af50-c77f9efb9986" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558860 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e95e0b-6c18-42af-a847-a25520a997ec" containerName="main" Oct 30 20:04:55.558874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.558865 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79fb900-6199-42f1-83c2-3dcdb1671412" containerName="payload-pre-processing" Oct 30 20:04:55.561998 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.561971 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.574462 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.574436 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:04:55.584327 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.584307 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bw5jw\"" Oct 30 20:04:55.587284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.587257 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:04:55.600822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.600791 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.600957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.600830 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.600957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.600908 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601107 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601004 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601107 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601044 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601107 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601103 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601161 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601189 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.601240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.601208 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmdn\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702209 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702249 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702267 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702290 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702320 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702345 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702363 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmdn\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702449 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702500 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702781 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702998 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702831 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.702998 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.702973 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.703141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.703037 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.703141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.703119 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.704778 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.704749 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.705221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.705054 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.719046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.719019 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.719193 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.719128 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmdn\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\") pod \"maas-default-gateway-openshift-default-6f54779779-vv8z5\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:55.877373 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:55.877290 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:56.201211 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.201178 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:04:56.202788 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:04:56.202761 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413d9c9_a94c_4b84_ad92_eee68b5beac4.slice/crio-6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987 WatchSource:0}: Error finding container 6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987: Status 404 returned error can't find the container with id 6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987 Oct 30 20:04:56.204988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.204954 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:04:56.205069 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.205017 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:04:56.205069 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.205043 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:04:56.627447 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.627397 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" event={"ID":"8413d9c9-a94c-4b84-ad92-eee68b5beac4","Type":"ContainerStarted","Data":"c595a274b8b270407bf2a10b7cbdb0f9b568f43d59ce9277a708854b5388bce5"} Oct 30 20:04:56.627794 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.627766 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" event={"ID":"8413d9c9-a94c-4b84-ad92-eee68b5beac4","Type":"ContainerStarted","Data":"6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987"} Oct 30 20:04:56.691058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.690992 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podStartSLOduration=1.690976684 podStartE2EDuration="1.690976684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:56.67793921 +0000 UTC m=+3604.389464577" watchObservedRunningTime="2026-10-30 20:04:56.690976684 +0000 UTC m=+3604.402502050" Oct 30 20:04:56.887178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:56.886420 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:04:57.792771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:57.792736 4990 scope.go:122] "RemoveContainer" containerID="2584cfa0c06fd3f5a1030362f35d088e46a81841ee41a974cfc371f3c443280f" Oct 30 20:04:57.802671 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:57.802650 4990 scope.go:122] "RemoveContainer" containerID="271f235074ad95c14d99490919d01c80c8a5ae5995b1a5ebce7bd8427a8f08a7" Oct 30 20:04:57.811208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:57.811185 4990 scope.go:122] "RemoveContainer" containerID="35f9924a20691a88b6a96c78c811e382b8d052b4c86ff2e1ee53ee23741a1f1d" Oct 30 20:04:57.893378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:57.893226 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:04:57.893378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:57.893297 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:04:58.889116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:58.879688 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:04:58.889116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:58.879749 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:04:59.884114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:59.882576 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:04:59.884114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:04:59.882726 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:05:00.883261 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:00.883217 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:05:00.883546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:00.883517 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:05:01.885120 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:01.879599 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:05:01.885120 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:01.879657 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:05:02.877748 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:02.877709 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-vv8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Oct 30 20:05:02.877984 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:02.877772 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Oct 30 20:05:03.641333 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:03.641290 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:04.422382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:04.422305 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:05:04.422668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:04.422632 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:05:04.423578 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:04.423557 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:05:05.094760 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:05:05.094724 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:05:08.526170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.526133 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn"] Oct 30 20:05:08.531284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.531261 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.555934 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.555907 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:05:08.556209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.556187 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:08.556344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.556239 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:05:08.556344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.556268 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:05:08.558446 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.558413 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn"] Oct 30 20:05:08.622496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.622456 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.622682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.622506 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp7p\" (UniqueName: \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.622682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.622525 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.723796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.723761 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp7p\" (UniqueName: \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.723796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.723798 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.724035 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.723861 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.724528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.724507 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.726613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.726594 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.741674 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.741648 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp7p\" (UniqueName: \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\") pod \"maas-api-revoke-sub-85237e45-w57kn\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:08.844312 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:08.844280 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:09.149902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:09.149828 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn"] Oct 30 20:05:09.153430 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:05:09.153399 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228f2260_f71b_41d9_b6cd_979724656dc5.slice/crio-988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8 WatchSource:0}: Error finding container 988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8: Status 404 returned error can't find the container with id 988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8 Oct 30 20:05:09.706722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:09.706685 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" event={"ID":"228f2260-f71b-41d9-b6cd-979724656dc5","Type":"ContainerStarted","Data":"13e30e6554509a59116d472ec45a18baa1d54a2211c74dd783737c8cd1bc14f0"} Oct 30 20:05:09.707139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:09.706738 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" event={"ID":"228f2260-f71b-41d9-b6cd-979724656dc5","Type":"ContainerStarted","Data":"988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8"} Oct 30 20:05:09.744280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:09.744178 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" podStartSLOduration=1.744164838 podStartE2EDuration="1.744164838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:09.741277629 +0000 UTC m=+3617.452802981" watchObservedRunningTime="2026-10-30 20:05:09.744164838 +0000 UTC m=+3617.455690204" Oct 30 20:05:10.605621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.605586 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:05:10.609140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.609118 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.637916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.637888 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:05:10.714566 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.714523 4990 generic.go:356] "Generic (PLEG): container finished" podID="228f2260-f71b-41d9-b6cd-979724656dc5" containerID="13e30e6554509a59116d472ec45a18baa1d54a2211c74dd783737c8cd1bc14f0" exitCode=0 Oct 30 20:05:10.714966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.714580 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" event={"ID":"228f2260-f71b-41d9-b6cd-979724656dc5","Type":"ContainerDied","Data":"13e30e6554509a59116d472ec45a18baa1d54a2211c74dd783737c8cd1bc14f0"} Oct 30 20:05:10.743063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743008 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8sk\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743072 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743161 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743193 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743226 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743217 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743326 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743354 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743375 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.743593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.743433 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844473 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8sk\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844521 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844554 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844581 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844719 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844854 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844827 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844865 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844898 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.844954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.844934 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.845161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.845061 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.845230 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.845182 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.845357 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.845309 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.845553 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.845532 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.845767 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.845741 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.846917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.846895 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.847145 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.847128 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.861843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.861772 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.862060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.862041 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8sk\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\") pod \"maas-default-gateway-openshift-default-6f54779779-czfvt\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:10.925927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:10.925888 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:11.245590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.245512 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:05:11.248988 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:05:11.248957 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb08303_e74d_4246_8124_5c716d720da5.slice/crio-fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6 WatchSource:0}: Error finding container fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6: Status 404 returned error can't find the container with id fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6 Oct 30 20:05:11.251079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.251048 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:05:11.251168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.251129 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:05:11.251168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.251158 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:05:11.734527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.734476 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" event={"ID":"fcb08303-e74d-4246-8124-5c716d720da5","Type":"ContainerStarted","Data":"7550ad647610d548a80f62fea9176bceb0271f8ab3167870642166d983485119"} Oct 30 20:05:11.735015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.734548 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" event={"ID":"fcb08303-e74d-4246-8124-5c716d720da5","Type":"ContainerStarted","Data":"fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6"} Oct 30 20:05:11.816299 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.814061 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podStartSLOduration=1.814040487 podStartE2EDuration="1.814040487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:11.79658739 +0000 UTC m=+3619.508112762" watchObservedRunningTime="2026-10-30 20:05:11.814040487 +0000 UTC m=+3619.525565854" Oct 30 20:05:11.926152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.926120 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:11.928914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:11.928889 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:12.057579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.057548 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\") pod \"228f2260-f71b-41d9-b6cd-979724656dc5\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " Oct 30 20:05:12.057783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.057610 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\") pod \"228f2260-f71b-41d9-b6cd-979724656dc5\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " Oct 30 20:05:12.057783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.057633 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\" (UniqueName: \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\") pod \"228f2260-f71b-41d9-b6cd-979724656dc5\" (UID: \"228f2260-f71b-41d9-b6cd-979724656dc5\") " Oct 30 20:05:12.058046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.058018 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca" pod "228f2260-f71b-41d9-b6cd-979724656dc5" (UID: "228f2260-f71b-41d9-b6cd-979724656dc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:12.059694 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.059674 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca" pod "228f2260-f71b-41d9-b6cd-979724656dc5" (UID: "228f2260-f71b-41d9-b6cd-979724656dc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:12.059765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.059724 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p" pod "228f2260-f71b-41d9-b6cd-979724656dc5" (UID: "228f2260-f71b-41d9-b6cd-979724656dc5"). InnerVolumeSpecName "kube-api-access-gvp7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:12.158513 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.158474 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:05:12.158513 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.158508 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228f2260-f71b-41d9-b6cd-979724656dc5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:05:12.158513 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.158517 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp7p\" (UniqueName: \"kubernetes.io/projected/228f2260-f71b-41d9-b6cd-979724656dc5-kube-api-access-gvp7p\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:05:12.741419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.741330 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" Oct 30 20:05:12.741419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.741366 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn" event={"ID":"228f2260-f71b-41d9-b6cd-979724656dc5","Type":"ContainerDied","Data":"988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8"} Oct 30 20:05:12.741950 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.741429 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="988a463d09b53d1a7ffbea612f1047a7c627db825a73eba1295cbb988c04e0b8" Oct 30 20:05:12.927754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.927716 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:12.927956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:12.927775 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:13.926471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:13.926434 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:13.926884 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:13.926502 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:14.927083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:14.927035 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:14.927629 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:14.927145 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:15.927015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:15.926923 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.927015 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:15.926981 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:16.926739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:16.926702 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.926933 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:16.926767 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:17.926441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:17.926401 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-czfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.926858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:17.926458 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 20:05:19.521737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:19.521685 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:19.522250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:19.522032 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:19.522878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:19.522855 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:05:20.151073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:20.151031 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:21.943435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:21.943398 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:35.095418 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:05:35.095383 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:05:46.312029 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:46.311963 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.020135 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:47.020077 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.692867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.692820 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r"] Oct 30 20:05:56.693430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.693222 4990 cpu_manager.go:414] "containerMap: removing container" podUID="228f2260-f71b-41d9-b6cd-979724656dc5" containerName="revoke-keys" Oct 30 20:05:56.693430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.693235 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228f2260-f71b-41d9-b6cd-979724656dc5" containerName="revoke-keys" Oct 30 20:05:56.693430 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.693354 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="228f2260-f71b-41d9-b6cd-979724656dc5" containerName="revoke-keys" Oct 30 20:05:56.697762 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.697735 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.712754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.712729 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:05:56.712906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.712759 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:05:56.722839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.722795 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:56.723000 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.722855 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:05:56.726335 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.726305 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r"] Oct 30 20:05:56.814870 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.814834 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pf4\" (UniqueName: \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.815110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.814890 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.815110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.815043 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.916166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.916124 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pf4\" (UniqueName: \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.916364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.916200 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.916364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.916253 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.916867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.916846 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.918682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.918653 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:56.933028 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:56.933002 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pf4\" (UniqueName: \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\") pod \"maas-api-revoke-sub-c1e271a9-gh64r\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:57.011131 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:57.011029 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:05:57.308297 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:57.308262 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r"] Oct 30 20:05:57.311853 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:05:57.311823 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911dedab_8c9d_4696_8fae_a76b440eb622.slice/crio-f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706 WatchSource:0}: Error finding container f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706: Status 404 returned error can't find the container with id f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706 Oct 30 20:05:57.929835 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:57.929760 4990 scope.go:122] "RemoveContainer" containerID="6a4b9f44030ac8f952b793fc8fe6d392b777631e43d838e2bb5e0ec0da803e1d" Oct 30 20:05:57.939442 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:57.939423 4990 scope.go:122] "RemoveContainer" containerID="61c1942ab63b44fb4e10fddff1ebdb7b313baeef9b79b0f88a0a06912f1c42cf" Oct 30 20:05:57.947933 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:57.947911 4990 scope.go:122] "RemoveContainer" containerID="321440e2e69fa7a1723c6ca9d5c7b02ef7ec70b5c5d0bf08855dd9b9058b5814" Oct 30 20:05:58.062052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:58.061989 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" event={"ID":"911dedab-8c9d-4696-8fae-a76b440eb622","Type":"ContainerStarted","Data":"0793ba0fbbb29278f0fa9898b126f53de2193a0d43ff3e4a564af3f3911dcea4"} Oct 30 20:05:58.062259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:58.062119 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" event={"ID":"911dedab-8c9d-4696-8fae-a76b440eb622","Type":"ContainerStarted","Data":"f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706"} Oct 30 20:05:58.096231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:58.096170 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" podStartSLOduration=2.096151703 podStartE2EDuration="2.096151703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:58.094327268 +0000 UTC m=+3665.805852634" watchObservedRunningTime="2026-10-30 20:05:58.096151703 +0000 UTC m=+3665.807677071" Oct 30 20:05:59.069218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:59.069182 4990 generic.go:356] "Generic (PLEG): container finished" podID="911dedab-8c9d-4696-8fae-a76b440eb622" containerID="0793ba0fbbb29278f0fa9898b126f53de2193a0d43ff3e4a564af3f3911dcea4" exitCode=0 Oct 30 20:05:59.069703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:05:59.069272 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" event={"ID":"911dedab-8c9d-4696-8fae-a76b440eb622","Type":"ContainerDied","Data":"0793ba0fbbb29278f0fa9898b126f53de2193a0d43ff3e4a564af3f3911dcea4"} Oct 30 20:06:00.206640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.206614 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:06:00.246572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.246540 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\") pod \"911dedab-8c9d-4696-8fae-a76b440eb622\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " Oct 30 20:06:00.246774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.246585 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\") pod \"911dedab-8c9d-4696-8fae-a76b440eb622\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " Oct 30 20:06:00.246774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.246675 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\" (UniqueName: \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\") pod \"911dedab-8c9d-4696-8fae-a76b440eb622\" (UID: \"911dedab-8c9d-4696-8fae-a76b440eb622\") " Oct 30 20:06:00.247100 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.247060 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca" pod "911dedab-8c9d-4696-8fae-a76b440eb622" (UID: "911dedab-8c9d-4696-8fae-a76b440eb622"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:00.248913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.248846 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4" pod "911dedab-8c9d-4696-8fae-a76b440eb622" (UID: "911dedab-8c9d-4696-8fae-a76b440eb622"). InnerVolumeSpecName "kube-api-access-p7pf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:00.249251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.248923 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca" pod "911dedab-8c9d-4696-8fae-a76b440eb622" (UID: "911dedab-8c9d-4696-8fae-a76b440eb622"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:00.347709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.347612 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pf4\" (UniqueName: \"kubernetes.io/projected/911dedab-8c9d-4696-8fae-a76b440eb622-kube-api-access-p7pf4\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:00.347709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.347649 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:00.347709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:00.347664 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911dedab-8c9d-4696-8fae-a76b440eb622-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:01.081603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:01.081573 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" Oct 30 20:06:01.081794 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:01.081613 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r" event={"ID":"911dedab-8c9d-4696-8fae-a76b440eb622","Type":"ContainerDied","Data":"f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706"} Oct 30 20:06:01.081794 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:01.081661 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f798756527ea0ae1b1a78609b9ccc4dc8b44e65d2149166620d0c79025156706" Oct 30 20:06:03.753059 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:03.753013 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:05.096228 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:06:05.096183 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:06:06.027321 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.027282 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw"] Oct 30 20:06:06.037608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.037575 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2"] Oct 30 20:06:06.039747 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.039720 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9457c9-j8gfw"] Oct 30 20:06:06.041979 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.041944 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06a6026c-9lgh2"] Oct 30 20:06:06.400141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.400102 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c10a45-480b-45d6-ab0b-7c412a77816b" path="/var/lib/kubelet/pods/a2c10a45-480b-45d6-ab0b-7c412a77816b/volumes" Oct 30 20:06:06.400576 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:06.400559 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca5b25c-a234-4c92-8685-ab02abd021f4" path="/var/lib/kubelet/pods/eca5b25c-a234-4c92-8685-ab02abd021f4/volumes" Oct 30 20:06:10.841328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.841293 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:06:10.841990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.841956 4990 cpu_manager.go:414] "containerMap: removing container" podUID="911dedab-8c9d-4696-8fae-a76b440eb622" containerName="revoke-keys" Oct 30 20:06:10.841990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.841981 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911dedab-8c9d-4696-8fae-a76b440eb622" containerName="revoke-keys" Oct 30 20:06:10.842196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.842125 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="911dedab-8c9d-4696-8fae-a76b440eb622" containerName="revoke-keys" Oct 30 20:06:10.846449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.846428 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.882683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.882648 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:06:10.943775 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943736 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.943975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943790 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.943975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943852 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.943975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943891 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.943975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943947 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.944217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.943983 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.944217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.944027 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.944217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.944085 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzq7c\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:10.944217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:10.944168 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.044991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.044947 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.044991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.044993 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzq7c\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045012 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045062 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045127 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045160 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045189 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045222 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045258 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045482 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045535 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045827 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.045964 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045837 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.046027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.045993 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.047504 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.047481 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.047604 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.047592 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.075554 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.075516 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.075785 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.075759 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzq7c\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\") pod \"maas-default-gateway-openshift-default-6f54779779-8l6gm\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.161689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.161602 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:11.506463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.506434 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:06:11.509224 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:06:11.509170 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d25c5e7_7376_49ac_bf09_25e92359d510.slice/crio-6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b WatchSource:0}: Error finding container 6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b: Status 404 returned error can't find the container with id 6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b Oct 30 20:06:11.511396 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.511360 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:06:11.511484 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.511438 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:06:11.511535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:11.511509 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:06:12.147976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.147933 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" event={"ID":"9d25c5e7-7376-49ac-bf09-25e92359d510","Type":"ContainerStarted","Data":"8873c9a7f3b46720983d5143633ddef5be5b51840989130e9b12988d4af7202a"} Oct 30 20:06:12.148627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.148001 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" event={"ID":"9d25c5e7-7376-49ac-bf09-25e92359d510","Type":"ContainerStarted","Data":"6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b"} Oct 30 20:06:12.162165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.162129 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:12.163874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.163847 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.164006 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.163894 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:12.220715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:12.220654 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podStartSLOduration=2.22063929 podStartE2EDuration="2.22063929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.208890579 +0000 UTC m=+3679.920415945" watchObservedRunningTime="2026-10-30 20:06:12.22063929 +0000 UTC m=+3679.932164655" Oct 30 20:06:13.161978 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:13.161942 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.162405 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:13.161995 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:14.162491 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:14.162450 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.164192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:14.162510 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:15.162110 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:15.162069 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.162309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:15.162144 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:16.162859 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:16.162819 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.163408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:16.162882 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:17.162160 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:17.162120 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.162386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:17.162195 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:18.162603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:18.162558 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.163030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:18.162631 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": dial tcp 10.134.1.86:15021: connect: connection refused" Oct 30 20:06:19.981506 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:19.981470 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8"] Oct 30 20:06:19.984078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:19.984059 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:19.996597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:19.996567 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:06:20.006630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.006573 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:06:20.006630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.006578 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:20.006906 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.006700 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:06:20.009609 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.009579 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8"] Oct 30 20:06:20.088007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.087957 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.088223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.088018 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.088223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.088116 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbnl\" (UniqueName: \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.163079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.163036 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:20.163244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.163141 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:20.189489 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.189458 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.189672 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.189497 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.189672 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.189515 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbnl\" (UniqueName: \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.190232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.190197 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.191887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.191862 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.206340 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.206312 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbnl\" (UniqueName: \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\") pod \"maas-api-revoke-sub-2e1f5b5f-b5dz8\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.298077 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.297994 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:20.598939 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:20.598905 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8"] Oct 30 20:06:20.603021 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:06:20.602990 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af375d2_036f_4859_8f64_b917b356a960.slice/crio-d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0 WatchSource:0}: Error finding container d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0: Status 404 returned error can't find the container with id d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0 Oct 30 20:06:21.205367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.205277 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:21.205367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.205334 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.231867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.231828 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" event={"ID":"1af375d2-036f-4859-8f64-b917b356a960","Type":"ContainerStarted","Data":"f3afeb5978058c995db11add27aa2a2b99dc3bce1e14b30eaab3889a4656f097"} Oct 30 20:06:21.232078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.231892 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" event={"ID":"1af375d2-036f-4859-8f64-b917b356a960","Type":"ContainerStarted","Data":"d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0"} Oct 30 20:06:21.265608 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.265545 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" podStartSLOduration=2.265526277 podStartE2EDuration="2.265526277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:21.264149462 +0000 UTC m=+3688.975674824" watchObservedRunningTime="2026-10-30 20:06:21.265526277 +0000 UTC m=+3688.977051645" Oct 30 20:06:21.900892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.900844 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:21.901308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.901287 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:21.902084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:21.902064 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:06:22.239708 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:22.239619 4990 generic.go:356] "Generic (PLEG): container finished" podID="1af375d2-036f-4859-8f64-b917b356a960" containerID="f3afeb5978058c995db11add27aa2a2b99dc3bce1e14b30eaab3889a4656f097" exitCode=0 Oct 30 20:06:22.240132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:22.239703 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" event={"ID":"1af375d2-036f-4859-8f64-b917b356a960","Type":"ContainerDied","Data":"f3afeb5978058c995db11add27aa2a2b99dc3bce1e14b30eaab3889a4656f097"} Oct 30 20:06:23.377766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.377737 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:23.524359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.524266 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\" (UniqueName: \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\") pod \"1af375d2-036f-4859-8f64-b917b356a960\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " Oct 30 20:06:23.524359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.524337 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\") pod \"1af375d2-036f-4859-8f64-b917b356a960\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " Oct 30 20:06:23.524567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.524433 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\") pod \"1af375d2-036f-4859-8f64-b917b356a960\" (UID: \"1af375d2-036f-4859-8f64-b917b356a960\") " Oct 30 20:06:23.524789 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.524746 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca" pod "1af375d2-036f-4859-8f64-b917b356a960" (UID: "1af375d2-036f-4859-8f64-b917b356a960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:23.526459 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.526436 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl" pod "1af375d2-036f-4859-8f64-b917b356a960" (UID: "1af375d2-036f-4859-8f64-b917b356a960"). InnerVolumeSpecName "kube-api-access-wxbnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:23.526568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.526553 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca" pod "1af375d2-036f-4859-8f64-b917b356a960" (UID: "1af375d2-036f-4859-8f64-b917b356a960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:23.625895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.625855 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbnl\" (UniqueName: \"kubernetes.io/projected/1af375d2-036f-4859-8f64-b917b356a960-kube-api-access-wxbnl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:23.625895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.625889 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af375d2-036f-4859-8f64-b917b356a960-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:23.625895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:23.625900 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af375d2-036f-4859-8f64-b917b356a960-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:24.253038 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:24.253007 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" Oct 30 20:06:24.253252 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:24.253049 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8" event={"ID":"1af375d2-036f-4859-8f64-b917b356a960","Type":"ContainerDied","Data":"d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0"} Oct 30 20:06:24.253252 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:24.253117 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d96c72ec05e0b0507b825fc6135a1dc7b3a5c4189d4e205a5f9970525666efb0" Oct 30 20:06:31.187083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.187046 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85"] Oct 30 20:06:31.187647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.187458 4990 cpu_manager.go:414] "containerMap: removing container" podUID="1af375d2-036f-4859-8f64-b917b356a960" containerName="revoke-keys" Oct 30 20:06:31.187647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.187471 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af375d2-036f-4859-8f64-b917b356a960" containerName="revoke-keys" Oct 30 20:06:31.187647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.187533 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af375d2-036f-4859-8f64-b917b356a960" containerName="revoke-keys" Oct 30 20:06:31.189633 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.189611 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.201996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.201959 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:06:31.211858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.211817 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:31.211858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.211842 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:06:31.212144 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.211951 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:06:31.215191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.215166 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85"] Oct 30 20:06:31.300721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.300683 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.300927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.300804 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.300927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.300885 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm6v\" (UniqueName: \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.402196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.402155 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.402196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.402196 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm6v\" (UniqueName: \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.402487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.402249 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.402959 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.402920 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.404542 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.404520 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.424803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.424774 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm6v\" (UniqueName: \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\") pod \"maas-api-revoke-sub-ec843018-8cs85\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.502322 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.502238 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:31.811658 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:31.811623 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85"] Oct 30 20:06:31.816553 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:06:31.816510 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14146f5d_3d57_457d_8ad1_7d19a8fe4e15.slice/crio-a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669 WatchSource:0}: Error finding container a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669: Status 404 returned error can't find the container with id a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669 Oct 30 20:06:32.303104 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:32.303055 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" event={"ID":"14146f5d-3d57-457d-8ad1-7d19a8fe4e15","Type":"ContainerStarted","Data":"32d5b8d593536cdfbb880ec3ee8f44f523c696af01f0f926c6771f42c97d46da"} Oct 30 20:06:32.303528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:32.303142 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" event={"ID":"14146f5d-3d57-457d-8ad1-7d19a8fe4e15","Type":"ContainerStarted","Data":"a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669"} Oct 30 20:06:32.337506 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:32.337450 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" podStartSLOduration=1.3374359710000001 podStartE2EDuration="1.337435971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:32.336280546 +0000 UTC m=+3700.047805913" watchObservedRunningTime="2026-10-30 20:06:32.337435971 +0000 UTC m=+3700.048961337" Oct 30 20:06:32.612752 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:06:32.612709 4990 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14146f5d_3d57_457d_8ad1_7d19a8fe4e15.slice/crio-conmon-32d5b8d593536cdfbb880ec3ee8f44f523c696af01f0f926c6771f42c97d46da.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:06:33.314805 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:33.314766 4990 generic.go:356] "Generic (PLEG): container finished" podID="14146f5d-3d57-457d-8ad1-7d19a8fe4e15" containerID="32d5b8d593536cdfbb880ec3ee8f44f523c696af01f0f926c6771f42c97d46da" exitCode=0 Oct 30 20:06:33.315358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:33.314867 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" event={"ID":"14146f5d-3d57-457d-8ad1-7d19a8fe4e15","Type":"ContainerDied","Data":"32d5b8d593536cdfbb880ec3ee8f44f523c696af01f0f926c6771f42c97d46da"} Oct 30 20:06:34.451046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.451021 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:34.549039 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.549006 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\") pod \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " Oct 30 20:06:34.549250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.549070 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\") pod \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " Oct 30 20:06:34.549250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.549158 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\" (UniqueName: \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\") pod \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\" (UID: \"14146f5d-3d57-457d-8ad1-7d19a8fe4e15\") " Oct 30 20:06:34.549482 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.549456 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca" pod "14146f5d-3d57-457d-8ad1-7d19a8fe4e15" (UID: "14146f5d-3d57-457d-8ad1-7d19a8fe4e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:34.551275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.551248 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca" pod "14146f5d-3d57-457d-8ad1-7d19a8fe4e15" (UID: "14146f5d-3d57-457d-8ad1-7d19a8fe4e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:34.551399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.551297 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v" pod "14146f5d-3d57-457d-8ad1-7d19a8fe4e15" (UID: "14146f5d-3d57-457d-8ad1-7d19a8fe4e15"). InnerVolumeSpecName "kube-api-access-9rm6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:34.650022 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.649938 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:34.650022 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.649971 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:34.650022 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:34.649982 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm6v\" (UniqueName: \"kubernetes.io/projected/14146f5d-3d57-457d-8ad1-7d19a8fe4e15-kube-api-access-9rm6v\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:06:35.095452 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:06:35.095416 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:06:35.327995 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:35.327964 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" Oct 30 20:06:35.328257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:35.327997 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85" event={"ID":"14146f5d-3d57-457d-8ad1-7d19a8fe4e15","Type":"ContainerDied","Data":"a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669"} Oct 30 20:06:35.328257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:35.328042 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53acfcc2d661984ff75c963f2500d8564d7b44adcca783ee641c6b2b8c68669" Oct 30 20:06:38.533305 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:38.533271 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.303079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:39.303041 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:57.978808 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:57.978774 4990 scope.go:122] "RemoveContainer" containerID="5cf8a0cfe05f60edf0d8f1c2b3721a5279b53ee464791332e0278cec396fa329" Oct 30 20:06:57.988021 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:06:57.988000 4990 scope.go:122] "RemoveContainer" containerID="1358b7d6f79a7aed170aaf1dd7b318b70416a6b32b5ab64decb60e25dc51097c" Oct 30 20:07:02.242946 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:02.242903 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.386871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:02.386826 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.878765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:03.878670 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:05.095262 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:07:05.095221 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:07:09.173227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.173194 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc"] Oct 30 20:07:09.173642 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.173568 4990 cpu_manager.go:414] "containerMap: removing container" podUID="14146f5d-3d57-457d-8ad1-7d19a8fe4e15" containerName="revoke-keys" Oct 30 20:07:09.173642 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.173580 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14146f5d-3d57-457d-8ad1-7d19a8fe4e15" containerName="revoke-keys" Oct 30 20:07:09.173718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.173667 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="14146f5d-3d57-457d-8ad1-7d19a8fe4e15" containerName="revoke-keys" Oct 30 20:07:09.175885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.175868 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.188498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.188474 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:07:09.198385 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.198353 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:07:09.198644 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.198361 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:09.198922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.198903 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:07:09.201220 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.201192 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc"] Oct 30 20:07:09.214220 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.214145 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.214367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.214220 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.214447 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.214368 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvqv\" (UniqueName: \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.314921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.314883 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvqv\" (UniqueName: \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.314921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.314923 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.315196 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.314972 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.315704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.315676 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.317309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.317289 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.331839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.331807 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvqv\" (UniqueName: \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\") pod \"maas-api-revoke-sub-046e1343-wvzcc\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.487834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.487751 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:09.783912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:09.783837 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc"] Oct 30 20:07:09.787514 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:07:09.787488 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8e9f49_8232_464c_8b90_9cef0723c0fe.slice/crio-5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf WatchSource:0}: Error finding container 5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf: Status 404 returned error can't find the container with id 5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf Oct 30 20:07:10.566666 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:10.566628 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" event={"ID":"8d8e9f49-8232-464c-8b90-9cef0723c0fe","Type":"ContainerStarted","Data":"2b0a74aa6d3310c29da5559733545c5ae263630dfe2234f388903f929871a5c6"} Oct 30 20:07:10.567368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:10.566710 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" event={"ID":"8d8e9f49-8232-464c-8b90-9cef0723c0fe","Type":"ContainerStarted","Data":"5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf"} Oct 30 20:07:10.602065 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:10.602022 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" podStartSLOduration=1.6020081130000001 podStartE2EDuration="1.602008113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:10.60070072 +0000 UTC m=+3738.312226086" watchObservedRunningTime="2026-10-30 20:07:10.602008113 +0000 UTC m=+3738.313533479" Oct 30 20:07:11.575995 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:11.575960 4990 generic.go:356] "Generic (PLEG): container finished" podID="8d8e9f49-8232-464c-8b90-9cef0723c0fe" containerID="2b0a74aa6d3310c29da5559733545c5ae263630dfe2234f388903f929871a5c6" exitCode=0 Oct 30 20:07:11.576510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:11.576050 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" event={"ID":"8d8e9f49-8232-464c-8b90-9cef0723c0fe","Type":"ContainerDied","Data":"2b0a74aa6d3310c29da5559733545c5ae263630dfe2234f388903f929871a5c6"} Oct 30 20:07:12.712081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.712056 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:12.746259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.746222 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\") pod \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " Oct 30 20:07:12.746445 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.746278 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\") pod \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " Oct 30 20:07:12.746445 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.746332 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\" (UniqueName: \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\") pod \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\" (UID: \"8d8e9f49-8232-464c-8b90-9cef0723c0fe\") " Oct 30 20:07:12.746640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.746613 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca" pod "8d8e9f49-8232-464c-8b90-9cef0723c0fe" (UID: "8d8e9f49-8232-464c-8b90-9cef0723c0fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:12.746753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.746714 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:12.749177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.749145 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca" pod "8d8e9f49-8232-464c-8b90-9cef0723c0fe" (UID: "8d8e9f49-8232-464c-8b90-9cef0723c0fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:12.749343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.749240 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv" pod "8d8e9f49-8232-464c-8b90-9cef0723c0fe" (UID: "8d8e9f49-8232-464c-8b90-9cef0723c0fe"). InnerVolumeSpecName "kube-api-access-7xvqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:12.847718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.847641 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d8e9f49-8232-464c-8b90-9cef0723c0fe-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:12.847718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:12.847669 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xvqv\" (UniqueName: \"kubernetes.io/projected/8d8e9f49-8232-464c-8b90-9cef0723c0fe-kube-api-access-7xvqv\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:13.588872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:13.588840 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" Oct 30 20:07:13.589060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:13.588875 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc" event={"ID":"8d8e9f49-8232-464c-8b90-9cef0723c0fe","Type":"ContainerDied","Data":"5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf"} Oct 30 20:07:13.589060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:13.588931 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d09057860a68de647964358530ea5006bf6ca69f53ff48499bfbe98617cc2cf" Oct 30 20:07:19.490990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.490952 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv"] Oct 30 20:07:19.491469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.491357 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8d8e9f49-8232-464c-8b90-9cef0723c0fe" containerName="revoke-keys" Oct 30 20:07:19.491469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.491369 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8e9f49-8232-464c-8b90-9cef0723c0fe" containerName="revoke-keys" Oct 30 20:07:19.491469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.491428 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8e9f49-8232-464c-8b90-9cef0723c0fe" containerName="revoke-keys" Oct 30 20:07:19.493566 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.493548 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.506132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.506103 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:07:19.515990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.515958 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:19.516199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.516003 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:07:19.516329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.516307 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:07:19.519558 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.519531 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv"] Oct 30 20:07:19.609529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.609493 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.609713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.609552 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99nr\" (UniqueName: \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.609757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.609713 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.711220 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.711181 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99nr\" (UniqueName: \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.711433 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.711252 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.711433 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.711304 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.712073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.712046 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.713814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.713779 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.729115 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.729076 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99nr\" (UniqueName: \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\") pod \"maas-api-revoke-sub-d3fba056-k2jlv\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:19.807713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:19.807672 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:20.112049 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:20.111966 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv"] Oct 30 20:07:20.115967 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:07:20.115931 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290e676c_8152_4cac_a74e_cc36dbc0125a.slice/crio-828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324 WatchSource:0}: Error finding container 828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324: Status 404 returned error can't find the container with id 828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324 Oct 30 20:07:20.631579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:20.631492 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" event={"ID":"290e676c-8152-4cac-a74e-cc36dbc0125a","Type":"ContainerStarted","Data":"6db6205d131d2dcefb50e8f7e2410bb51bbb588a5b408d749ef3c426f99b6f72"} Oct 30 20:07:20.631579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:20.631551 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" event={"ID":"290e676c-8152-4cac-a74e-cc36dbc0125a","Type":"ContainerStarted","Data":"828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324"} Oct 30 20:07:20.669925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:20.669876 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" podStartSLOduration=1.6698618170000001 podStartE2EDuration="1.669861817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:20.669029673 +0000 UTC m=+3748.380555038" watchObservedRunningTime="2026-10-30 20:07:20.669861817 +0000 UTC m=+3748.381387162" Oct 30 20:07:21.649677 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:21.649636 4990 generic.go:356] "Generic (PLEG): container finished" podID="290e676c-8152-4cac-a74e-cc36dbc0125a" containerID="6db6205d131d2dcefb50e8f7e2410bb51bbb588a5b408d749ef3c426f99b6f72" exitCode=0 Oct 30 20:07:21.650831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:21.650665 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" event={"ID":"290e676c-8152-4cac-a74e-cc36dbc0125a","Type":"ContainerDied","Data":"6db6205d131d2dcefb50e8f7e2410bb51bbb588a5b408d749ef3c426f99b6f72"} Oct 30 20:07:23.052036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.049186 4990 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8l6gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:07:23.052036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.049253 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:23.062116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.059937 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:23.208937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.208905 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\") pod \"290e676c-8152-4cac-a74e-cc36dbc0125a\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " Oct 30 20:07:23.209194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.208959 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\" (UniqueName: \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\") pod \"290e676c-8152-4cac-a74e-cc36dbc0125a\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " Oct 30 20:07:23.209194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.209062 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\") pod \"290e676c-8152-4cac-a74e-cc36dbc0125a\" (UID: \"290e676c-8152-4cac-a74e-cc36dbc0125a\") " Oct 30 20:07:23.209358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.209332 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca" pod "290e676c-8152-4cac-a74e-cc36dbc0125a" (UID: "290e676c-8152-4cac-a74e-cc36dbc0125a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:23.211726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.211692 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca" pod "290e676c-8152-4cac-a74e-cc36dbc0125a" (UID: "290e676c-8152-4cac-a74e-cc36dbc0125a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:23.211843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.211732 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr" pod "290e676c-8152-4cac-a74e-cc36dbc0125a" (UID: "290e676c-8152-4cac-a74e-cc36dbc0125a"). InnerVolumeSpecName "kube-api-access-n99nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:23.310579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.310537 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:23.310759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.310587 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290e676c-8152-4cac-a74e-cc36dbc0125a-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:23.310759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.310606 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n99nr\" (UniqueName: \"kubernetes.io/projected/290e676c-8152-4cac-a74e-cc36dbc0125a-kube-api-access-n99nr\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:23.668842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.668747 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" Oct 30 20:07:23.668842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.668780 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv" event={"ID":"290e676c-8152-4cac-a74e-cc36dbc0125a","Type":"ContainerDied","Data":"828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324"} Oct 30 20:07:23.668842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:23.668842 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828a8b02ddf2fdc3f38c218b7e8a232744c0ea671c13f79fd2a906db3d435324" Oct 30 20:07:35.095724 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:07:35.095685 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:07:42.097796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.097750 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x"] Oct 30 20:07:42.098352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.098137 4990 cpu_manager.go:414] "containerMap: removing container" podUID="290e676c-8152-4cac-a74e-cc36dbc0125a" containerName="revoke-keys" Oct 30 20:07:42.098352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.098150 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290e676c-8152-4cac-a74e-cc36dbc0125a" containerName="revoke-keys" Oct 30 20:07:42.098352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.098219 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="290e676c-8152-4cac-a74e-cc36dbc0125a" containerName="revoke-keys" Oct 30 20:07:42.101325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.101300 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.114457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.114435 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:07:42.116230 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.116202 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:07:42.124684 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.124659 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:42.126284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.126263 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:07:42.128323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.128299 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x"] Oct 30 20:07:42.207962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.207926 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.208179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.207983 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.208179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.208084 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mzk\" (UniqueName: \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.309033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.308995 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.309219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.309053 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.309219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.309073 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mzk\" (UniqueName: \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.309814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.309791 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.311484 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.311464 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.326374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.326352 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mzk\" (UniqueName: \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\") pod \"maas-api-revoke-sub-ae594979-d6x9x\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.418606 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.418534 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:42.722221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.722190 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x"] Oct 30 20:07:42.723925 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:07:42.723895 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda99b164_605e_465c_893a_a747e2c8c165.slice/crio-caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76 WatchSource:0}: Error finding container caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76: Status 404 returned error can't find the container with id caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76 Oct 30 20:07:42.812824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:42.812777 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" event={"ID":"da99b164-605e-465c-893a-a747e2c8c165","Type":"ContainerStarted","Data":"caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76"} Oct 30 20:07:43.820219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:43.820183 4990 generic.go:356] "Generic (PLEG): container finished" podID="da99b164-605e-465c-893a-a747e2c8c165" containerID="05b25c790ec1a7b578017d05d4bd4603f08a66bc14a612646a1fe4d4becdf8bf" exitCode=0 Oct 30 20:07:43.820644 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:43.820266 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" event={"ID":"da99b164-605e-465c-893a-a747e2c8c165","Type":"ContainerDied","Data":"05b25c790ec1a7b578017d05d4bd4603f08a66bc14a612646a1fe4d4becdf8bf"} Oct 30 20:07:44.955194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:44.955170 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:45.033546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.033512 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\" (UniqueName: \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\") pod \"da99b164-605e-465c-893a-a747e2c8c165\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " Oct 30 20:07:45.033712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.033622 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\") pod \"da99b164-605e-465c-893a-a747e2c8c165\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " Oct 30 20:07:45.033712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.033650 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\") pod \"da99b164-605e-465c-893a-a747e2c8c165\" (UID: \"da99b164-605e-465c-893a-a747e2c8c165\") " Oct 30 20:07:45.034002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.033978 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca" pod "da99b164-605e-465c-893a-a747e2c8c165" (UID: "da99b164-605e-465c-893a-a747e2c8c165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:45.035800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.035770 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk" pod "da99b164-605e-465c-893a-a747e2c8c165" (UID: "da99b164-605e-465c-893a-a747e2c8c165"). InnerVolumeSpecName "kube-api-access-b9mzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:45.035931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.035808 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca" pod "da99b164-605e-465c-893a-a747e2c8c165" (UID: "da99b164-605e-465c-893a-a747e2c8c165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:45.134348 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.134273 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9mzk\" (UniqueName: \"kubernetes.io/projected/da99b164-605e-465c-893a-a747e2c8c165-kube-api-access-b9mzk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:45.134348 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.134299 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da99b164-605e-465c-893a-a747e2c8c165-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:45.134348 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.134310 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da99b164-605e-465c-893a-a747e2c8c165-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:07:45.833596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.833565 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" Oct 30 20:07:45.833774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.833566 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x" event={"ID":"da99b164-605e-465c-893a-a747e2c8c165","Type":"ContainerDied","Data":"caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76"} Oct 30 20:07:45.833774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:45.833694 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caa0ae82296777ebb6f0c961df272f614d9ef9b58aa7b1e2648fa4fcf5798c76" Oct 30 20:07:48.024188 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:48.024152 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj"] Oct 30 20:07:48.033572 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:48.033542 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2430d912-r5zcj"] Oct 30 20:07:48.399355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:48.399321 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e1f9a2-a64e-4964-9245-1a8d7d58f82c" path="/var/lib/kubelet/pods/46e1f9a2-a64e-4964-9245-1a8d7d58f82c/volumes" Oct 30 20:07:58.020489 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.020451 4990 scope.go:122] "RemoveContainer" containerID="75a60e5fda899f31f0f5517b6a88cb8e651059d37859dc4cbe01125bab31396c" Oct 30 20:07:58.057681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.057654 4990 scope.go:122] "RemoveContainer" containerID="acc6d2c94ae72906d8ef715a1d61bd4097707a98562c94bac70297b0f403b933" Oct 30 20:07:58.089477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.089450 4990 scope.go:122] "RemoveContainer" containerID="c7b6c339a305d967e395ec9e7cf05dd43c2e21e0e4c782772aafbe791cca49a7" Oct 30 20:07:58.123356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.120701 4990 scope.go:122] "RemoveContainer" containerID="61afc38526cff51a8fa9435de7d146dbb582472ab7cd2cac7c0cfe07adb0e655" Oct 30 20:07:58.154981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.154958 4990 scope.go:122] "RemoveContainer" containerID="04488aa46046fc6e1be67f73ffab9a9509fc01bae3e573a3fb49b81cb05acc3a" Oct 30 20:07:58.178313 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.178292 4990 scope.go:122] "RemoveContainer" containerID="3b202356aaec9518363c87eb7e135aaffdf45e3958abccfe7988f2ae5240be37" Oct 30 20:07:58.208252 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.208117 4990 scope.go:122] "RemoveContainer" containerID="4b1eaabb3ba05ae6ff206bf945d4eed9c81a766bafe8ce00f2980d2175d4d615" Oct 30 20:07:58.250933 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:07:58.250800 4990 scope.go:122] "RemoveContainer" containerID="c05978f903f9a54e8b8c7787aaa640d644a11f5c6ddccdeabdbf5fedc67eca03" Oct 30 20:08:02.647415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:02.647376 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:03.604309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.604273 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8"] Oct 30 20:08:03.604668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.604654 4990 cpu_manager.go:414] "containerMap: removing container" podUID="da99b164-605e-465c-893a-a747e2c8c165" containerName="revoke-keys" Oct 30 20:08:03.604719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.604668 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da99b164-605e-465c-893a-a747e2c8c165" containerName="revoke-keys" Oct 30 20:08:03.604763 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.604748 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="da99b164-605e-465c-893a-a747e2c8c165" containerName="revoke-keys" Oct 30 20:08:03.607727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.607710 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.620541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.620517 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:08:03.630449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.630421 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:03.630449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.630441 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:08:03.630449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.630434 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:08:03.633034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.633009 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8"] Oct 30 20:08:03.738351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.738319 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnzb\" (UniqueName: \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.738756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.738368 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.738756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.738503 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.839153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.839116 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.839361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.839183 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnzb\" (UniqueName: \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.839361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.839304 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.839914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.839891 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.841593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.841572 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.857019 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.856958 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnzb\" (UniqueName: \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\") pod \"maas-api-revoke-sub-a4aef7d7-2szs8\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.920360 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.920323 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:03.992221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:03.992186 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:04.173908 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:04.173811 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:04.217151 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:04.217118 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8"] Oct 30 20:08:04.220910 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:08:04.220882 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfb09ab_b805_4bc5_92a2_4b5185de5ba9.slice/crio-ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5 WatchSource:0}: Error finding container ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5: Status 404 returned error can't find the container with id ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5 Oct 30 20:08:04.984693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:04.984654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" event={"ID":"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9","Type":"ContainerStarted","Data":"2e317035f6c97eacc38b999e2430575f6b50a51f4a7be011aa8ac2c869f98628"} Oct 30 20:08:04.985129 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:04.984712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" event={"ID":"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9","Type":"ContainerStarted","Data":"ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5"} Oct 30 20:08:05.019246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:05.019186 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" podStartSLOduration=2.019171248 podStartE2EDuration="2.019171248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:05.016903512 +0000 UTC m=+3792.728428872" watchObservedRunningTime="2026-10-30 20:08:05.019171248 +0000 UTC m=+3792.730696617" Oct 30 20:08:05.096270 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:08:05.096237 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:08:05.991731 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:05.991695 4990 generic.go:356] "Generic (PLEG): container finished" podID="7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" containerID="2e317035f6c97eacc38b999e2430575f6b50a51f4a7be011aa8ac2c869f98628" exitCode=0 Oct 30 20:08:05.992145 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:05.991756 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" event={"ID":"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9","Type":"ContainerDied","Data":"2e317035f6c97eacc38b999e2430575f6b50a51f4a7be011aa8ac2c869f98628"} Oct 30 20:08:07.125266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.125244 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:07.269534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.269454 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\" (UniqueName: \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\") pod \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " Oct 30 20:08:07.269534 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.269495 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\") pod \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " Oct 30 20:08:07.269736 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.269537 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\") pod \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\" (UID: \"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9\") " Oct 30 20:08:07.269951 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.269927 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca" pod "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" (UID: "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:07.271673 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.271642 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca" pod "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" (UID: "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:07.271673 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.271648 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb" pod "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" (UID: "7cfb09ab-b805-4bc5-92a2-4b5185de5ba9"). InnerVolumeSpecName "kube-api-access-2pnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:07.371407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.371360 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnzb\" (UniqueName: \"kubernetes.io/projected/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-kube-api-access-2pnzb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:07.371407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.371403 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:07.371407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:07.371416 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:08.003773 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:08.003739 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" Oct 30 20:08:08.003958 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:08.003776 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8" event={"ID":"7cfb09ab-b805-4bc5-92a2-4b5185de5ba9","Type":"ContainerDied","Data":"ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5"} Oct 30 20:08:08.003958 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:08.003827 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9818a75a17eba1b2163f46af827c639b5bea5ec7de4a84e8d643d06d6b04b5" Oct 30 20:08:16.456387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:16.456345 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:16.588537 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:16.588496 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:20.029617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:20.029577 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd"] Oct 30 20:08:20.038183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:20.038151 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f1b11d-w98sd"] Oct 30 20:08:20.401668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:20.401617 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d227471-b67a-4e3f-8225-f26fd36709a5" path="/var/lib/kubelet/pods/7d227471-b67a-4e3f-8225-f26fd36709a5/volumes" Oct 30 20:08:24.781107 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.781053 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf"] Oct 30 20:08:24.781589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.781450 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" containerName="revoke-keys" Oct 30 20:08:24.781589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.781465 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" containerName="revoke-keys" Oct 30 20:08:24.781589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.781546 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" containerName="revoke-keys" Oct 30 20:08:24.785725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.785704 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.798570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.798549 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:08:24.808498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.808467 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:24.808620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.808495 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:08:24.808779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.808764 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:08:24.811223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.811201 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf"] Oct 30 20:08:24.882766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.882689 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.882766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.882723 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4m8\" (UniqueName: \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.882953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.882788 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.983581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.983543 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.983581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.983580 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4m8\" (UniqueName: \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.983806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.983622 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.984379 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.984358 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:24.985903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:24.985871 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:25.000612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:25.000584 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4m8\" (UniqueName: \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\") pod \"maas-api-revoke-sub-79805ca9-wvhdf\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:25.098874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:25.098841 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:25.395879 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:25.395852 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf"] Oct 30 20:08:25.398736 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:08:25.398661 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64346a5f_7b5f_4054_9742_d77152586b6f.slice/crio-099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954 WatchSource:0}: Error finding container 099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954: Status 404 returned error can't find the container with id 099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954 Oct 30 20:08:26.136113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:26.136050 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" event={"ID":"64346a5f-7b5f-4054-9742-d77152586b6f","Type":"ContainerStarted","Data":"793076f37db1baf6a404eb6263537ed748f676858e347af5d1377e68f982c29e"} Oct 30 20:08:26.136503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:26.136131 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" event={"ID":"64346a5f-7b5f-4054-9742-d77152586b6f","Type":"ContainerStarted","Data":"099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954"} Oct 30 20:08:26.171751 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:26.171692 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" podStartSLOduration=2.171678803 podStartE2EDuration="2.171678803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:26.170360784 +0000 UTC m=+3813.881886150" watchObservedRunningTime="2026-10-30 20:08:26.171678803 +0000 UTC m=+3813.883204169" Oct 30 20:08:27.142475 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:27.142435 4990 generic.go:356] "Generic (PLEG): container finished" podID="64346a5f-7b5f-4054-9742-d77152586b6f" containerID="793076f37db1baf6a404eb6263537ed748f676858e347af5d1377e68f982c29e" exitCode=0 Oct 30 20:08:27.142877 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:27.142524 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" event={"ID":"64346a5f-7b5f-4054-9742-d77152586b6f","Type":"ContainerDied","Data":"793076f37db1baf6a404eb6263537ed748f676858e347af5d1377e68f982c29e"} Oct 30 20:08:28.277486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.277462 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:28.317441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.317405 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\") pod \"64346a5f-7b5f-4054-9742-d77152586b6f\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " Oct 30 20:08:28.317621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.317490 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\") pod \"64346a5f-7b5f-4054-9742-d77152586b6f\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " Oct 30 20:08:28.317621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.317564 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\" (UniqueName: \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\") pod \"64346a5f-7b5f-4054-9742-d77152586b6f\" (UID: \"64346a5f-7b5f-4054-9742-d77152586b6f\") " Oct 30 20:08:28.317912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.317886 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca" pod "64346a5f-7b5f-4054-9742-d77152586b6f" (UID: "64346a5f-7b5f-4054-9742-d77152586b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:28.320060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.320027 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca" pod "64346a5f-7b5f-4054-9742-d77152586b6f" (UID: "64346a5f-7b5f-4054-9742-d77152586b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:28.320201 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.320034 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8" pod "64346a5f-7b5f-4054-9742-d77152586b6f" (UID: "64346a5f-7b5f-4054-9742-d77152586b6f"). InnerVolumeSpecName "kube-api-access-gb4m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:28.418777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.418743 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4m8\" (UniqueName: \"kubernetes.io/projected/64346a5f-7b5f-4054-9742-d77152586b6f-kube-api-access-gb4m8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:28.418777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.418773 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:28.418952 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:28.418787 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64346a5f-7b5f-4054-9742-d77152586b6f-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:29.154894 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:29.154867 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" Oct 30 20:08:29.155077 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:29.154862 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf" event={"ID":"64346a5f-7b5f-4054-9742-d77152586b6f","Type":"ContainerDied","Data":"099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954"} Oct 30 20:08:29.155077 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:29.155023 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="099a2a89b792ef5bdcf308fc48726b48f7e93e3d2f6d7641ff43b41e97797954" Oct 30 20:08:29.917700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:29.917666 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:30.066375 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:30.066340 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:35.096682 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:08:35.096638 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:08:45.838477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.838439 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v"] Oct 30 20:08:45.839118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.838876 4990 cpu_manager.go:414] "containerMap: removing container" podUID="64346a5f-7b5f-4054-9742-d77152586b6f" containerName="revoke-keys" Oct 30 20:08:45.839118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.838894 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64346a5f-7b5f-4054-9742-d77152586b6f" containerName="revoke-keys" Oct 30 20:08:45.839118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.839042 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="64346a5f-7b5f-4054-9742-d77152586b6f" containerName="revoke-keys" Oct 30 20:08:45.842222 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.842194 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:45.867460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.867429 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:08:45.867681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.867534 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:08:45.867681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.867582 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:45.867681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.867664 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:08:45.871697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.871660 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v"] Oct 30 20:08:45.899962 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.899928 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:45.900241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.899975 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbskl\" (UniqueName: \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:45.900241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:45.900011 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.001474 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.001437 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.001659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.001555 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.001659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.001596 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbskl\" (UniqueName: \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.002280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.002247 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.003865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.003844 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.018937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.018914 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbskl\" (UniqueName: \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\") pod \"maas-api-revoke-sub-0b7e8edb-9fd4v\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.156254 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.156172 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:46.452665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:46.452584 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v"] Oct 30 20:08:46.455652 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:08:46.455620 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f20d754_43ad_4494_b96c_26f8901d0af1.slice/crio-f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6 WatchSource:0}: Error finding container f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6: Status 404 returned error can't find the container with id f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6 Oct 30 20:08:47.275250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:47.275215 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" event={"ID":"0f20d754-43ad-4494-b96c-26f8901d0af1","Type":"ContainerStarted","Data":"53b39fd6cce423a19e03e8be480a125acea7d90ab95977622f7158711f5905f9"} Oct 30 20:08:47.275648 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:47.275292 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" event={"ID":"0f20d754-43ad-4494-b96c-26f8901d0af1","Type":"ContainerStarted","Data":"f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6"} Oct 30 20:08:47.310058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:47.310007 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" podStartSLOduration=2.309993946 podStartE2EDuration="2.309993946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:47.308800521 +0000 UTC m=+3835.020325881" watchObservedRunningTime="2026-10-30 20:08:47.309993946 +0000 UTC m=+3835.021519312" Oct 30 20:08:48.281916 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:48.281882 4990 generic.go:356] "Generic (PLEG): container finished" podID="0f20d754-43ad-4494-b96c-26f8901d0af1" containerID="53b39fd6cce423a19e03e8be480a125acea7d90ab95977622f7158711f5905f9" exitCode=0 Oct 30 20:08:48.282329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:48.281975 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" event={"ID":"0f20d754-43ad-4494-b96c-26f8901d0af1","Type":"ContainerDied","Data":"53b39fd6cce423a19e03e8be480a125acea7d90ab95977622f7158711f5905f9"} Oct 30 20:08:49.414126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.414080 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:49.533233 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.533197 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\" (UniqueName: \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\") pod \"0f20d754-43ad-4494-b96c-26f8901d0af1\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " Oct 30 20:08:49.533401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.533337 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\") pod \"0f20d754-43ad-4494-b96c-26f8901d0af1\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " Oct 30 20:08:49.533401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.533363 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\") pod \"0f20d754-43ad-4494-b96c-26f8901d0af1\" (UID: \"0f20d754-43ad-4494-b96c-26f8901d0af1\") " Oct 30 20:08:49.533762 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.533733 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca" pod "0f20d754-43ad-4494-b96c-26f8901d0af1" (UID: "0f20d754-43ad-4494-b96c-26f8901d0af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:49.535363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.535341 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl" pod "0f20d754-43ad-4494-b96c-26f8901d0af1" (UID: "0f20d754-43ad-4494-b96c-26f8901d0af1"). InnerVolumeSpecName "kube-api-access-fbskl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:49.535460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.535364 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca" pod "0f20d754-43ad-4494-b96c-26f8901d0af1" (UID: "0f20d754-43ad-4494-b96c-26f8901d0af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:49.634300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.634270 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:49.634300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.634295 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d754-43ad-4494-b96c-26f8901d0af1-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:49.634300 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:49.634305 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbskl\" (UniqueName: \"kubernetes.io/projected/0f20d754-43ad-4494-b96c-26f8901d0af1-kube-api-access-fbskl\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:50.295436 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:50.295403 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" Oct 30 20:08:50.295436 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:50.295428 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v" event={"ID":"0f20d754-43ad-4494-b96c-26f8901d0af1","Type":"ContainerDied","Data":"f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6"} Oct 30 20:08:50.295664 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:50.295475 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ddce203c857f2fae7e648353a2fe0992b6bc7fbdfad49d27094aeb0aed4cc6" Oct 30 20:08:56.597899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.597848 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt"] Oct 30 20:08:56.598574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.598264 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0f20d754-43ad-4494-b96c-26f8901d0af1" containerName="revoke-keys" Oct 30 20:08:56.598574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.598277 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f20d754-43ad-4494-b96c-26f8901d0af1" containerName="revoke-keys" Oct 30 20:08:56.598574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.598341 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f20d754-43ad-4494-b96c-26f8901d0af1" containerName="revoke-keys" Oct 30 20:08:56.601798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.601779 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.617030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.617002 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:56.617182 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.617062 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:08:56.620318 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.620292 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt"] Oct 30 20:08:56.678102 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.678058 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:08:56.678319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.678058 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:08:56.693486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.693456 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.693650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.693491 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzlh\" (UniqueName: \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.693650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.693532 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.794546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.794500 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.794546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.794541 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzlh\" (UniqueName: \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.794811 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.794583 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.795318 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.795296 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.797136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.797079 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.811649 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.811618 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzlh\" (UniqueName: \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\") pod \"maas-api-revoke-sub-3d4156f1-dzclt\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:56.919206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:56.919129 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:57.217429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:57.217400 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt"] Oct 30 20:08:57.220306 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:08:57.220262 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1bb518_7b58_45ea_900f_c455507842ca.slice/crio-cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e WatchSource:0}: Error finding container cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e: Status 404 returned error can't find the container with id cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e Oct 30 20:08:57.338435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:57.338401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" event={"ID":"8d1bb518-7b58-45ea-900f-c455507842ca","Type":"ContainerStarted","Data":"cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e"} Oct 30 20:08:58.346800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.346707 4990 generic.go:356] "Generic (PLEG): container finished" podID="8d1bb518-7b58-45ea-900f-c455507842ca" containerID="37d392e7602a00eb3715112626c0a55a4c97acf53a4a324c652301b967a0d9cd" exitCode=0 Oct 30 20:08:58.347275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.346791 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" event={"ID":"8d1bb518-7b58-45ea-900f-c455507842ca","Type":"ContainerDied","Data":"37d392e7602a00eb3715112626c0a55a4c97acf53a4a324c652301b967a0d9cd"} Oct 30 20:08:58.379005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.378976 4990 scope.go:122] "RemoveContainer" containerID="704e49f8acd478cc32221db8000679e140cc208c1836e57d2fb341d28d7d789c" Oct 30 20:08:58.388958 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.388935 4990 scope.go:122] "RemoveContainer" containerID="e731195a47100de4d09b14ddc095be9b7a9e53aa87f7f183025c7d2e3aea0eee" Oct 30 20:08:58.397931 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.397915 4990 scope.go:122] "RemoveContainer" containerID="8c27bb4e5f952da0ea78967e3952c35f66cdf39942b03860feba88810fce9380" Oct 30 20:08:58.406232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.406211 4990 scope.go:122] "RemoveContainer" containerID="c760cb7cd2b258e090d47ea6481c029a7f996fbb506693cc2d50e4fcf9071338" Oct 30 20:08:58.413976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.413955 4990 scope.go:122] "RemoveContainer" containerID="17c22b4ae2c50c4893eac2cc5ee1d8d7dd24b884c5f830df0043baee9682fdde" Oct 30 20:08:58.423617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:58.423600 4990 scope.go:122] "RemoveContainer" containerID="2c2d7255e8dd4eca316ef63ce5298d69c9da50c5ad33cbfcef19548f0c8068b4" Oct 30 20:08:59.489779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.489756 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:08:59.620766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.620678 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\") pod \"8d1bb518-7b58-45ea-900f-c455507842ca\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " Oct 30 20:08:59.620766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.620713 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\" (UniqueName: \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\") pod \"8d1bb518-7b58-45ea-900f-c455507842ca\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " Oct 30 20:08:59.620766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.620756 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\") pod \"8d1bb518-7b58-45ea-900f-c455507842ca\" (UID: \"8d1bb518-7b58-45ea-900f-c455507842ca\") " Oct 30 20:08:59.621252 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.621229 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca" pod "8d1bb518-7b58-45ea-900f-c455507842ca" (UID: "8d1bb518-7b58-45ea-900f-c455507842ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:59.622875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.622849 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh" pod "8d1bb518-7b58-45ea-900f-c455507842ca" (UID: "8d1bb518-7b58-45ea-900f-c455507842ca"). InnerVolumeSpecName "kube-api-access-kpzlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:59.622875 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.622861 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca" pod "8d1bb518-7b58-45ea-900f-c455507842ca" (UID: "8d1bb518-7b58-45ea-900f-c455507842ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:59.722315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.722273 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:59.722315 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.722311 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1bb518-7b58-45ea-900f-c455507842ca-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:08:59.722564 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:08:59.722326 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzlh\" (UniqueName: \"kubernetes.io/projected/8d1bb518-7b58-45ea-900f-c455507842ca-kube-api-access-kpzlh\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:00.358626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:00.358596 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" Oct 30 20:09:00.358626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:00.358622 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt" event={"ID":"8d1bb518-7b58-45ea-900f-c455507842ca","Type":"ContainerDied","Data":"cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e"} Oct 30 20:09:00.358851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:00.358665 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdea132b557747a0fd86e1b2ba232470bf630334f9b09c9c2d9adda65c1af76e" Oct 30 20:09:04.120638 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:04.120597 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:04.305896 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:04.305862 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:05.095393 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:09:05.095345 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:09:10.032123 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:10.027077 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:17.989907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:17.989861 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt"] Oct 30 20:09:17.999350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:17.999319 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dzclt"] Oct 30 20:09:18.037344 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.037309 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7"] Oct 30 20:09:18.037746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.037734 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8d1bb518-7b58-45ea-900f-c455507842ca" containerName="revoke-keys" Oct 30 20:09:18.037797 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.037749 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1bb518-7b58-45ea-900f-c455507842ca" containerName="revoke-keys" Oct 30 20:09:18.037856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.037848 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1bb518-7b58-45ea-900f-c455507842ca" containerName="revoke-keys" Oct 30 20:09:18.041285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.041262 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.053764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.053742 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:09:18.053764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.053723 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:09:18.053922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.053742 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:09:18.053922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.053763 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:09:18.060132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.060043 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7"] Oct 30 20:09:18.116959 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.116929 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.117148 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.117073 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.117204 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.117169 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k99k\" (UniqueName: \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.217817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.217785 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.217980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.217879 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.217980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.217909 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k99k\" (UniqueName: \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.218576 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.218551 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.220166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.220147 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.234625 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.234600 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k99k\" (UniqueName: \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\") pod \"maas-api-revoke-sub-3d4156f1-zdxs7\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.354008 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.353982 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:18.400868 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.400842 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1bb518-7b58-45ea-900f-c455507842ca" path="/var/lib/kubelet/pods/8d1bb518-7b58-45ea-900f-c455507842ca/volumes" Oct 30 20:09:18.650284 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:18.650205 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7"] Oct 30 20:09:18.653428 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:09:18.653399 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5717c0e_1314_4f12_9c6f_6182683d0172.slice/crio-aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7 WatchSource:0}: Error finding container aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7: Status 404 returned error can't find the container with id aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7 Oct 30 20:09:19.034907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:19.034820 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4"] Oct 30 20:09:19.037188 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:19.037164 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b180a7e9-cpps4"] Oct 30 20:09:19.498086 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:19.498050 4990 generic.go:356] "Generic (PLEG): container finished" podID="a5717c0e-1314-4f12-9c6f-6182683d0172" containerID="7f5ae8ceb116e43b4d5b79d48a228226b7400fad9fd54ba5951f690e437d7abc" exitCode=0 Oct 30 20:09:19.498268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:19.498146 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" event={"ID":"a5717c0e-1314-4f12-9c6f-6182683d0172","Type":"ContainerDied","Data":"7f5ae8ceb116e43b4d5b79d48a228226b7400fad9fd54ba5951f690e437d7abc"} Oct 30 20:09:19.498268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:19.498203 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" event={"ID":"a5717c0e-1314-4f12-9c6f-6182683d0172","Type":"ContainerStarted","Data":"aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7"} Oct 30 20:09:20.399357 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.399324 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a" path="/var/lib/kubelet/pods/9a14c8c3-d0c2-48f0-9f9d-0e7cef498d5a/volumes" Oct 30 20:09:20.634047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.634023 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:20.740067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.739992 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\") pod \"a5717c0e-1314-4f12-9c6f-6182683d0172\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " Oct 30 20:09:20.740239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.740073 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\" (UniqueName: \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\") pod \"a5717c0e-1314-4f12-9c6f-6182683d0172\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " Oct 30 20:09:20.740239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.740172 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\") pod \"a5717c0e-1314-4f12-9c6f-6182683d0172\" (UID: \"a5717c0e-1314-4f12-9c6f-6182683d0172\") " Oct 30 20:09:20.740456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.740412 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca" pod "a5717c0e-1314-4f12-9c6f-6182683d0172" (UID: "a5717c0e-1314-4f12-9c6f-6182683d0172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:20.740549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.740524 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:20.742135 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.742113 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca" pod "a5717c0e-1314-4f12-9c6f-6182683d0172" (UID: "a5717c0e-1314-4f12-9c6f-6182683d0172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:20.742230 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.742186 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k" pod "a5717c0e-1314-4f12-9c6f-6182683d0172" (UID: "a5717c0e-1314-4f12-9c6f-6182683d0172"). InnerVolumeSpecName "kube-api-access-7k99k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:20.841328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.841299 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5717c0e-1314-4f12-9c6f-6182683d0172-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:20.841328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:20.841331 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k99k\" (UniqueName: \"kubernetes.io/projected/a5717c0e-1314-4f12-9c6f-6182683d0172-kube-api-access-7k99k\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:21.512263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:21.512233 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" Oct 30 20:09:21.512263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:21.512254 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7" event={"ID":"a5717c0e-1314-4f12-9c6f-6182683d0172","Type":"ContainerDied","Data":"aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7"} Oct 30 20:09:21.512725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:21.512303 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2af22022cb97a50fe8eb6a9dec4c5eccfbeeb9d51956d5ab568d5620d83ad7" Oct 30 20:09:35.095277 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:09:35.095191 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:09:38.690527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:38.690495 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:40.584411 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:40.584371 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:51.968047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:51.968009 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7"] Oct 30 20:09:51.977505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:51.977478 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zdxs7"] Oct 30 20:09:52.023404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.023374 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw"] Oct 30 20:09:52.023734 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.023721 4990 cpu_manager.go:414] "containerMap: removing container" podUID="a5717c0e-1314-4f12-9c6f-6182683d0172" containerName="revoke-keys" Oct 30 20:09:52.023779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.023735 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5717c0e-1314-4f12-9c6f-6182683d0172" containerName="revoke-keys" Oct 30 20:09:52.023816 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.023807 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5717c0e-1314-4f12-9c6f-6182683d0172" containerName="revoke-keys" Oct 30 20:09:52.026746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.026729 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.039263 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.039238 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:09:52.039688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.039670 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:09:52.039733 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.039713 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:09:52.040216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.040201 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:09:52.052690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.052665 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw"] Oct 30 20:09:52.131462 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.131428 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.131645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.131480 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.131645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.131587 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdc8c\" (UniqueName: \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.232310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.232237 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.232310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.232279 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdc8c\" (UniqueName: \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.232507 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.232340 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.232891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.232860 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.234588 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.234567 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.256922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.256891 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdc8c\" (UniqueName: \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\") pod \"maas-api-revoke-sub-3d4156f1-z7zdw\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.340367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.340342 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:52.400068 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.400039 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5717c0e-1314-4f12-9c6f-6182683d0172" path="/var/lib/kubelet/pods/a5717c0e-1314-4f12-9c6f-6182683d0172/volumes" Oct 30 20:09:52.638422 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.638390 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw"] Oct 30 20:09:52.641933 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:09:52.641902 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deec254_75fd_4163_8c05_6d40ccbdb0fa.slice/crio-4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368 WatchSource:0}: Error finding container 4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368: Status 404 returned error can't find the container with id 4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368 Oct 30 20:09:52.686582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:52.686549 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" event={"ID":"4deec254-75fd-4163-8c05-6d40ccbdb0fa","Type":"ContainerStarted","Data":"4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368"} Oct 30 20:09:53.693612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:53.693534 4990 generic.go:356] "Generic (PLEG): container finished" podID="4deec254-75fd-4163-8c05-6d40ccbdb0fa" containerID="0bac6ebbad7b4aa4676bc739c8af65f86fe31fae425bb79d8fae1563e29b1682" exitCode=0 Oct 30 20:09:53.693997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:53.693621 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" event={"ID":"4deec254-75fd-4163-8c05-6d40ccbdb0fa","Type":"ContainerDied","Data":"0bac6ebbad7b4aa4676bc739c8af65f86fe31fae425bb79d8fae1563e29b1682"} Oct 30 20:09:54.747949 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.747922 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:09:54.763681 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.758797 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:09:54.788111 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.788067 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:09:54.798158 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.798134 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:09:54.840016 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.839995 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:54.956065 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.955995 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\") pod \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " Oct 30 20:09:54.956236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.956085 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\") pod \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " Oct 30 20:09:54.956236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.956149 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\" (UniqueName: \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\") pod \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\" (UID: \"4deec254-75fd-4163-8c05-6d40ccbdb0fa\") " Oct 30 20:09:54.956374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.956351 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca" pod "4deec254-75fd-4163-8c05-6d40ccbdb0fa" (UID: "4deec254-75fd-4163-8c05-6d40ccbdb0fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:54.958280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.958246 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c" pod "4deec254-75fd-4163-8c05-6d40ccbdb0fa" (UID: "4deec254-75fd-4163-8c05-6d40ccbdb0fa"). InnerVolumeSpecName "kube-api-access-xdc8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:54.958401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:54.958283 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca" pod "4deec254-75fd-4163-8c05-6d40ccbdb0fa" (UID: "4deec254-75fd-4163-8c05-6d40ccbdb0fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:55.057516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.057482 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:55.057516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.057513 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4deec254-75fd-4163-8c05-6d40ccbdb0fa-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:55.057701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.057523 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdc8c\" (UniqueName: \"kubernetes.io/projected/4deec254-75fd-4163-8c05-6d40ccbdb0fa-kube-api-access-xdc8c\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:09:55.705880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.705843 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" event={"ID":"4deec254-75fd-4163-8c05-6d40ccbdb0fa","Type":"ContainerDied","Data":"4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368"} Oct 30 20:09:55.705880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.705869 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw" Oct 30 20:09:55.706125 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:55.705892 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7caa97a4f59e1144970d4023e90f5d7bb1978280050efd3ac18c8d40c9c368" Oct 30 20:09:58.466597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:58.466564 4990 scope.go:122] "RemoveContainer" containerID="562731f4c20d183de1d802448c170649be001206d0a21e90e1d1d0e9a4ae6914" Oct 30 20:09:58.475756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:58.475735 4990 scope.go:122] "RemoveContainer" containerID="853a003a6feec157bffcc001444a9ae4bb08ee00c5cf1c58730a3ba68388143b" Oct 30 20:09:58.483776 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:09:58.483757 4990 scope.go:122] "RemoveContainer" containerID="7ba26d37ff387a4ddda640a3dddb7b3359fd0184ae6f12f49dd3e32042478d97" Oct 30 20:10:04.249795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:04.249755 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:05.096169 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:10:05.096129 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:10:12.025593 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:12.025552 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn"] Oct 30 20:10:12.035192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:12.035160 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-w57kn"] Oct 30 20:10:12.399582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:12.399544 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228f2260-f71b-41d9-b6cd-979724656dc5" path="/var/lib/kubelet/pods/228f2260-f71b-41d9-b6cd-979724656dc5/volumes" Oct 30 20:10:13.667726 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:13.667687 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:14.112259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.112227 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:10:14.112756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.112729 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4deec254-75fd-4163-8c05-6d40ccbdb0fa" containerName="revoke-keys" Oct 30 20:10:14.112756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.112755 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deec254-75fd-4163-8c05-6d40ccbdb0fa" containerName="revoke-keys" Oct 30 20:10:14.112969 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.112865 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deec254-75fd-4163-8c05-6d40ccbdb0fa" containerName="revoke-keys" Oct 30 20:10:14.116438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.116412 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.133995 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.133970 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-013647-openshift-default-dockercfg-7cs9h\"" Oct 30 20:10:14.160199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.160164 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:10:14.206042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206005 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206044 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206282 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206068 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206282 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206208 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206286 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rw8n\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206394 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206438 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206455 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.206495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.206479 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.307785 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.307735 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.307798 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.307823 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.307856 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.307907 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308060 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308134 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308166 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308208 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rw8n\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308272 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308411 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308489 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.308712 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308684 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.309010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.308989 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.310787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.310764 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.310787 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.310780 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.325147 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.325116 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.325257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.325196 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rw8n\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\") pod \"e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.431433 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.431352 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:14.747625 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.747538 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:10:14.750693 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:10:14.750665 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423b2907_a3d9_4df1_b889_f7c725d4c9ae.slice/crio-ffcdff9397c550e8266843b1f74b32d1f56676cbd84023750fd5fb8081af3379 WatchSource:0}: Error finding container ffcdff9397c550e8266843b1f74b32d1f56676cbd84023750fd5fb8081af3379: Status 404 returned error can't find the container with id ffcdff9397c550e8266843b1f74b32d1f56676cbd84023750fd5fb8081af3379 Oct 30 20:10:14.752989 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.752952 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:14.753067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.753031 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:14.753067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.753062 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:14.814030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:14.814001 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerStarted","Data":"ffcdff9397c550e8266843b1f74b32d1f56676cbd84023750fd5fb8081af3379"} Oct 30 20:10:15.476721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:15.476681 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:15.822363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:15.822321 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerStarted","Data":"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec"} Oct 30 20:10:15.871574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:15.871520 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podStartSLOduration=1.871507456 podStartE2EDuration="1.871507456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:15.867955092 +0000 UTC m=+3923.579480458" watchObservedRunningTime="2026-10-30 20:10:15.871507456 +0000 UTC m=+3923.583032821" Oct 30 20:10:16.432559 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:16.432525 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:16.438118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:16.438078 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:16.829581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:16.829547 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:16.830548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:16.830524 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:18.296122 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:18.296060 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:18.451700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:18.451657 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.721443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:19.721403 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.872308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:19.872269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:20.057975 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.057931 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:20.126670 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.126638 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:10:20.130402 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.130386 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.143170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.143143 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-013647\"" Oct 30 20:10:20.155635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.155611 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:10:20.165963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.165926 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.166079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.165981 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxrp\" (UniqueName: \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.197555 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.197526 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:10:20.201084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.201064 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.209986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.209961 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:20.219246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.219228 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b210437a-dockercfg-hwhn6\"" Oct 30 20:10:20.222600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.222576 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:10:20.266670 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.266634 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.266880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.266856 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6d69\" (UniqueName: \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.266941 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.266914 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.267020 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.266990 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxrp\" (UniqueName: \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.267741 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.267713 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.287055 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.287025 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxrp\" (UniqueName: \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\") pod \"payload-processing-e2e-worker-single-main-013647-6f98d977494lzg\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.368575 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.368489 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.368575 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.368545 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6d69\" (UniqueName: \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.369250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.369230 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.404364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.404337 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6d69\" (UniqueName: \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\") pod \"payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.443167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.443138 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.514416 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.514383 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:20.750046 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.750016 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:10:20.752286 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:10:20.752252 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3af0089_ee49_4bea_85d8_5e9890425f77.slice/crio-281eee449dc9a25a89734a6b56998b8a7b51d2531fd187cdf6df1bd46413ef0f WatchSource:0}: Error finding container 281eee449dc9a25a89734a6b56998b8a7b51d2531fd187cdf6df1bd46413ef0f: Status 404 returned error can't find the container with id 281eee449dc9a25a89734a6b56998b8a7b51d2531fd187cdf6df1bd46413ef0f Oct 30 20:10:20.827561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.827536 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:10:20.831323 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:10:20.831290 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808c09c_783a_4a2e_b1c7_b3aa84852b9a.slice/crio-0e60fb85d6a67294ff692428b71f8694b371c6891fa275c6e8e4f397f27e5751 WatchSource:0}: Error finding container 0e60fb85d6a67294ff692428b71f8694b371c6891fa275c6e8e4f397f27e5751: Status 404 returned error can't find the container with id 0e60fb85d6a67294ff692428b71f8694b371c6891fa275c6e8e4f397f27e5751 Oct 30 20:10:20.862042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.862003 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" event={"ID":"c808c09c-783a-4a2e-b1c7-b3aa84852b9a","Type":"ContainerStarted","Data":"0e60fb85d6a67294ff692428b71f8694b371c6891fa275c6e8e4f397f27e5751"} Oct 30 20:10:20.864178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.864143 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" event={"ID":"d3af0089-ee49-4bea-85d8-5e9890425f77","Type":"ContainerStarted","Data":"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea"} Oct 30 20:10:20.864297 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.864204 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" event={"ID":"d3af0089-ee49-4bea-85d8-5e9890425f77","Type":"ContainerStarted","Data":"281eee449dc9a25a89734a6b56998b8a7b51d2531fd187cdf6df1bd46413ef0f"} Oct 30 20:10:20.864359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.864340 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:20.898963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:20.898921 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" podStartSLOduration=0.898909168 podStartE2EDuration="898.909168ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:20.89835146 +0000 UTC m=+3928.609876832" watchObservedRunningTime="2026-10-30 20:10:20.898909168 +0000 UTC m=+3928.610434533" Oct 30 20:10:21.625115 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:21.624996 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:21.871813 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:21.871769 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" event={"ID":"c808c09c-783a-4a2e-b1c7-b3aa84852b9a","Type":"ContainerStarted","Data":"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f"} Oct 30 20:10:21.872272 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:21.871861 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:21.926443 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:21.926358 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" podStartSLOduration=1.926344498 podStartE2EDuration="1.926344498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:21.924690959 +0000 UTC m=+3929.636216324" watchObservedRunningTime="2026-10-30 20:10:21.926344498 +0000 UTC m=+3929.637869863" Oct 30 20:10:23.540800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:23.540761 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.160927 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:25.160888 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:26.740081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:26.740037 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:28.587113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:28.583410 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:30.229102 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:30.229048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.864239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:31.864058 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.876442 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:31.876416 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:10:32.559897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.559049 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": read tcp 10.134.0.2:47092->10.134.1.98:15021: read: connection reset by peer" start-of-body= Oct 30 20:10:32.559897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.559136 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": read tcp 10.134.0.2:47092->10.134.1.98:15021: read: connection reset by peer" Oct 30 20:10:32.880551 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.880477 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:10:32.946394 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.946359 4990 generic.go:356] "Generic (PLEG): container finished" podID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerID="e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec" exitCode=0 Oct 30 20:10:32.946568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.946436 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerDied","Data":"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec"} Oct 30 20:10:32.946816 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.946802 4990 scope.go:122] "RemoveContainer" containerID="e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec" Oct 30 20:10:32.948028 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.947998 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:32.948175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.948083 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:32.948175 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:32.948153 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:10:33.244849 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:33.244751 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.517039 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:33.516951 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.924754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:33.924719 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.953973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:33.953943 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerStarted","Data":"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc"} Oct 30 20:10:34.085112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:34.082623 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:34.431784 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:34.431753 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:34.431986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:34.431931 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 20:10:34.432066 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:34.431987 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 20:10:35.113736 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:10:35.113695 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:10:35.395354 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:35.395184 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:35.432085 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:35.432050 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 20:10:35.432258 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:35.432121 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 20:10:35.548034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:35.547994 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.432179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:36.432145 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 20:10:36.432674 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:36.432205 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 20:10:36.913764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:36.913730 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:37.431953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:37.431916 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 20:10:37.432164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:37.431972 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 20:10:38.439113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:38.438484 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Oct 30 20:10:38.439113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:38.438550 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Oct 30 20:10:38.654086 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:38.654042 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:39.506973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:39.506937 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:39.507665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:39.507378 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:39.509168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:39.509145 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:10:40.341872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:40.341814 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:42.061997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:42.061957 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:43.643206 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:43.643170 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.243124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:45.242779 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:47.028376 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:47.028338 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:48.685080 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:48.685044 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:50.263909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:50.263847 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:51.925537 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:51.925500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.514342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:53.514304 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:55.067547 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:55.067506 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.836035 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:56.835991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.386260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:58.386224 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.514646 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:58.514613 4990 scope.go:122] "RemoveContainer" containerID="8b3f1b9492c5963affaff3a806666b70274d157ee0f2e261bedf89efc8b851b3" Oct 30 20:10:58.523603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:10:58.523583 4990 scope.go:122] "RemoveContainer" containerID="41ee65de0f76aeb6b6e9266c5200d80a2f6580a599e880da4649244bd668077c" Oct 30 20:11:00.032403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:00.032357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:01.026536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:01.026504 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r"] Oct 30 20:11:01.035746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:01.035723 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gh64r"] Oct 30 20:11:01.911113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:01.911048 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:02.399478 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:02.399434 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911dedab-8c9d-4696-8fae-a76b440eb622" path="/var/lib/kubelet/pods/911dedab-8c9d-4696-8fae-a76b440eb622/volumes" Oct 30 20:11:03.593117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:03.593062 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:04.366627 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:04.366549 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.095762 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:05.095728 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:11:05.305744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:05.305703 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:06.966075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:06.966036 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:08.687795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:08.687755 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:09.120918 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.120878 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq"] Oct 30 20:11:09.123479 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.123455 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.135821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.135798 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:09.145499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.145476 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:09.145621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.145595 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:11:09.145621 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.145598 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:09.148727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.148701 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq"] Oct 30 20:11:09.247301 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.247269 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs52r\" (UniqueName: \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.247461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.247322 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.247510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.247471 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.347928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.347896 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.348124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.347943 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs52r\" (UniqueName: \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.348124 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.347978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.348571 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.348547 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.350438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.350419 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.364842 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.364813 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs52r\" (UniqueName: \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\") pod \"maas-api-revoke-sub-70c748c1-j22pq\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.435442 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.435387 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:09.733493 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:09.733467 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq"] Oct 30 20:11:09.736359 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:11:09.736330 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf926b88b_ecae_4291_9803_4635c70dec74.slice/crio-561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6 WatchSource:0}: Error finding container 561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6: Status 404 returned error can't find the container with id 561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6 Oct 30 20:11:10.200029 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:10.199989 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" event={"ID":"f926b88b-ecae-4291-9803-4635c70dec74","Type":"ContainerStarted","Data":"ea62a110d04180e1b2f4c9c64fb2a964e82de874bf7aef29a7fd9510d3dc68b1"} Oct 30 20:11:10.200260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:10.200085 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" event={"ID":"f926b88b-ecae-4291-9803-4635c70dec74","Type":"ContainerStarted","Data":"561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6"} Oct 30 20:11:10.235125 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:10.235032 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" podStartSLOduration=1.235014302 podStartE2EDuration="1.235014302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:10.233318857 +0000 UTC m=+3977.944844223" watchObservedRunningTime="2026-10-30 20:11:10.235014302 +0000 UTC m=+3977.946539668" Oct 30 20:11:10.345521 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:10.345426 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:11.206660 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:11.206624 4990 generic.go:356] "Generic (PLEG): container finished" podID="f926b88b-ecae-4291-9803-4635c70dec74" containerID="ea62a110d04180e1b2f4c9c64fb2a964e82de874bf7aef29a7fd9510d3dc68b1" exitCode=0 Oct 30 20:11:11.207119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:11.206712 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" event={"ID":"f926b88b-ecae-4291-9803-4635c70dec74","Type":"ContainerDied","Data":"ea62a110d04180e1b2f4c9c64fb2a964e82de874bf7aef29a7fd9510d3dc68b1"} Oct 30 20:11:11.882033 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:11.881708 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.345228 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.345204 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:12.478691 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.478618 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\") pod \"f926b88b-ecae-4291-9803-4635c70dec74\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " Oct 30 20:11:12.478837 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.478792 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\" (UniqueName: \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\") pod \"f926b88b-ecae-4291-9803-4635c70dec74\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " Oct 30 20:11:12.478880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.478834 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\") pod \"f926b88b-ecae-4291-9803-4635c70dec74\" (UID: \"f926b88b-ecae-4291-9803-4635c70dec74\") " Oct 30 20:11:12.479241 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.479212 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca" pod "f926b88b-ecae-4291-9803-4635c70dec74" (UID: "f926b88b-ecae-4291-9803-4635c70dec74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:12.480687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.480664 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca" pod "f926b88b-ecae-4291-9803-4635c70dec74" (UID: "f926b88b-ecae-4291-9803-4635c70dec74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:12.480892 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.480876 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r" pod "f926b88b-ecae-4291-9803-4635c70dec74" (UID: "f926b88b-ecae-4291-9803-4635c70dec74"). InnerVolumeSpecName "kube-api-access-hs52r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:12.580404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.580368 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs52r\" (UniqueName: \"kubernetes.io/projected/f926b88b-ecae-4291-9803-4635c70dec74-kube-api-access-hs52r\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:12.580404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.580393 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:12.580404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:12.580403 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926b88b-ecae-4291-9803-4635c70dec74-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:13.219563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:13.219530 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" Oct 30 20:11:13.219753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:13.219559 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq" event={"ID":"f926b88b-ecae-4291-9803-4635c70dec74","Type":"ContainerDied","Data":"561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6"} Oct 30 20:11:13.219753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:13.219611 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561bd5f6364ce490cf6c9093c26fb666bdd8642a02b14c48e4ca2305a9e183d6" Oct 30 20:11:13.524231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:13.524085 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:15.113408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:15.113370 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:16.928543 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:16.928500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:18.533317 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:18.533268 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.244415 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.244373 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.378668 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.378621 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv"] Oct 30 20:11:20.379250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.379230 4990 cpu_manager.go:414] "containerMap: removing container" podUID="f926b88b-ecae-4291-9803-4635c70dec74" containerName="revoke-keys" Oct 30 20:11:20.379342 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.379252 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f926b88b-ecae-4291-9803-4635c70dec74" containerName="revoke-keys" Oct 30 20:11:20.379412 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.379359 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="f926b88b-ecae-4291-9803-4635c70dec74" containerName="revoke-keys" Oct 30 20:11:20.383309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.383285 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.400756 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.400702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.406618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.406593 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:20.406768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.406617 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:11:20.406768 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.406625 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:20.406965 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.406659 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:20.410574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.410550 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv"] Oct 30 20:11:20.455822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.455793 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctg6q\" (UniqueName: \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.455965 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.455844 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.455965 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.455864 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.557058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.557031 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.557273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.557073 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.557273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.557237 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctg6q\" (UniqueName: \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.557878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.557855 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.559531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.559508 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.574869 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.574842 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctg6q\" (UniqueName: \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.697199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.697156 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:20.999746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:20.999663 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv"] Oct 30 20:11:21.003859 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:11:21.003829 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adeea96_47d5_447f_ab29_e1d2e5219eb4.slice/crio-78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5 WatchSource:0}: Error finding container 78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5: Status 404 returned error can't find the container with id 78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5 Oct 30 20:11:21.268310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:21.268269 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" event={"ID":"2adeea96-47d5-447f-ab29-e1d2e5219eb4","Type":"ContainerStarted","Data":"78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5"} Oct 30 20:11:22.220469 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:22.220419 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.275796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:22.275760 4990 generic.go:356] "Generic (PLEG): container finished" podID="2adeea96-47d5-447f-ab29-e1d2e5219eb4" containerID="7e905d378c40376bcb0d10f744ba27f884416ed7c13145c991341de742ad37e6" exitCode=0 Oct 30 20:11:22.276228 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:22.275850 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" event={"ID":"2adeea96-47d5-447f-ab29-e1d2e5219eb4","Type":"ContainerDied","Data":"7e905d378c40376bcb0d10f744ba27f884416ed7c13145c991341de742ad37e6"} Oct 30 20:11:23.414818 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.414796 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:23.484709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.484669 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\") pod \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " Oct 30 20:11:23.484914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.484726 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\" (UniqueName: \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\") pod \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " Oct 30 20:11:23.484914 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.484815 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\") pod \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\" (UID: \"2adeea96-47d5-447f-ab29-e1d2e5219eb4\") " Oct 30 20:11:23.485169 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.485144 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca" pod "2adeea96-47d5-447f-ab29-e1d2e5219eb4" (UID: "2adeea96-47d5-447f-ab29-e1d2e5219eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:23.486867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.486833 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca" pod "2adeea96-47d5-447f-ab29-e1d2e5219eb4" (UID: "2adeea96-47d5-447f-ab29-e1d2e5219eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:23.487013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.486947 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q" pod "2adeea96-47d5-447f-ab29-e1d2e5219eb4" (UID: "2adeea96-47d5-447f-ab29-e1d2e5219eb4"). InnerVolumeSpecName "kube-api-access-ctg6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:23.586184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.586149 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:23.586184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.586174 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adeea96-47d5-447f-ab29-e1d2e5219eb4-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:23.586184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:23.586186 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctg6q\" (UniqueName: \"kubernetes.io/projected/2adeea96-47d5-447f-ab29-e1d2e5219eb4-kube-api-access-ctg6q\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:24.004563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.004471 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:24.025574 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.025536 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8"] Oct 30 20:11:24.034104 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.034067 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b5dz8"] Oct 30 20:11:24.289895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.289805 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" event={"ID":"2adeea96-47d5-447f-ab29-e1d2e5219eb4","Type":"ContainerDied","Data":"78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5"} Oct 30 20:11:24.289895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.289845 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv" Oct 30 20:11:24.289895 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.289882 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78461aae4cb411a3afdc931cb907d19516a368dd5740c802daf31be3760520c5" Oct 30 20:11:24.400535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:24.400498 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af375d2-036f-4859-8f64-b917b356a960" path="/var/lib/kubelet/pods/1af375d2-036f-4859-8f64-b917b356a960/volumes" Oct 30 20:11:25.272569 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:25.272526 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.579626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:25.579593 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.172065 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:27.172027 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:28.826004 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:28.825965 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:30.467524 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:30.467460 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:30.886218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:30.886185 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:11:30.920436 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:30.920410 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:11:31.081783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:31.081730 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.215968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:31.215864 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:32.015445 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:32.015394 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.473737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:33.473698 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.963391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:33.963351 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.038643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:34.038611 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv"] Oct 30 20:11:34.047739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:34.047713 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-013647-js4hv"] Oct 30 20:11:34.109352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:34.109321 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.400085 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:34.400050 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adeea96-47d5-447f-ab29-e1d2e5219eb4" path="/var/lib/kubelet/pods/2adeea96-47d5-447f-ab29-e1d2e5219eb4/volumes" Oct 30 20:11:34.623304 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:34.623267 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.025871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:35.025837 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85"] Oct 30 20:11:35.042231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:35.038822 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8cs85"] Oct 30 20:11:35.095988 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:35.095953 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:11:35.886990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:35.886941 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" containerID="cri-o://cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea" gracePeriod=25 Oct 30 20:11:35.920749 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:35.920683 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" containerID="cri-o://53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f" gracePeriod=25 Oct 30 20:11:36.400706 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:36.400674 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14146f5d-3d57-457d-8ad1-7d19a8fe4e15" path="/var/lib/kubelet/pods/14146f5d-3d57-457d-8ad1-7d19a8fe4e15/volumes" Oct 30 20:11:40.550419 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:40.550381 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.713221 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:40.713185 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.850159 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:40.850066 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:11:40.850387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:40.850359 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" containerID="cri-o://ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc" gracePeriod=30 Oct 30 20:11:40.924355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:40.924306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.874191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:41.874141 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:42.152685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.152593 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw"] Oct 30 20:11:42.161837 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.161808 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7zdw"] Oct 30 20:11:42.202258 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.202230 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7"] Oct 30 20:11:42.202688 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.202673 4990 cpu_manager.go:414] "containerMap: removing container" podUID="2adeea96-47d5-447f-ab29-e1d2e5219eb4" containerName="revoke-keys" Oct 30 20:11:42.202732 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.202689 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adeea96-47d5-447f-ab29-e1d2e5219eb4" containerName="revoke-keys" Oct 30 20:11:42.202769 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.202758 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adeea96-47d5-447f-ab29-e1d2e5219eb4" containerName="revoke-keys" Oct 30 20:11:42.206765 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.206748 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.222311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.222289 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:42.222423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.222380 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:11:42.222423 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.222387 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:11:42.223720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.223701 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:11:42.224776 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.224752 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7"] Oct 30 20:11:42.364899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.364862 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.364899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.364902 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.365171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.365020 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4tx\" (UniqueName: \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.400746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.400707 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deec254-75fd-4163-8c05-6d40ccbdb0fa" path="/var/lib/kubelet/pods/4deec254-75fd-4163-8c05-6d40ccbdb0fa/volumes" Oct 30 20:11:42.466082 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.466005 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4tx\" (UniqueName: \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.466280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.466129 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.466280 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.466172 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.466772 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.466739 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.468722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.468691 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.485034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.485006 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4tx\" (UniqueName: \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\") pod \"maas-api-revoke-sub-3d4156f1-l8dq7\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.519153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.519129 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:42.822996 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.822964 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7"] Oct 30 20:11:42.823874 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:11:42.823841 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e38ba93_ce7b_4613_8759_6d29cfaa2dac.slice/crio-3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107 WatchSource:0}: Error finding container 3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107: Status 404 returned error can't find the container with id 3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107 Oct 30 20:11:42.878058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:42.878029 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:43.404251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:43.404163 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" event={"ID":"3e38ba93-ce7b-4613-8759-6d29cfaa2dac","Type":"ContainerStarted","Data":"99bf873d39d2cef68444e1f997c4ab655ecc3b1237ac630a82a9c6fb80e74907"} Oct 30 20:11:43.404251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:43.404232 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" event={"ID":"3e38ba93-ce7b-4613-8759-6d29cfaa2dac","Type":"ContainerStarted","Data":"3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107"} Oct 30 20:11:43.445380 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:43.445329 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" podStartSLOduration=1.445316472 podStartE2EDuration="1.445316472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:43.442513055 +0000 UTC m=+4011.154038419" watchObservedRunningTime="2026-10-30 20:11:43.445316472 +0000 UTC m=+4011.156841858" Oct 30 20:11:44.411165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:44.411130 4990 generic.go:356] "Generic (PLEG): container finished" podID="3e38ba93-ce7b-4613-8759-6d29cfaa2dac" containerID="99bf873d39d2cef68444e1f997c4ab655ecc3b1237ac630a82a9c6fb80e74907" exitCode=0 Oct 30 20:11:44.411587 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:44.411208 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" event={"ID":"3e38ba93-ce7b-4613-8759-6d29cfaa2dac","Type":"ContainerDied","Data":"99bf873d39d2cef68444e1f997c4ab655ecc3b1237ac630a82a9c6fb80e74907"} Oct 30 20:11:45.548268 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.548244 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:45.697899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.697822 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\" (UniqueName: \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\") pod \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " Oct 30 20:11:45.698084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.697901 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\") pod \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " Oct 30 20:11:45.698084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.697927 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\") pod \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\" (UID: \"3e38ba93-ce7b-4613-8759-6d29cfaa2dac\") " Oct 30 20:11:45.698320 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.698297 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca" pod "3e38ba93-ce7b-4613-8759-6d29cfaa2dac" (UID: "3e38ba93-ce7b-4613-8759-6d29cfaa2dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:45.699984 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.699962 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx" pod "3e38ba93-ce7b-4613-8759-6d29cfaa2dac" (UID: "3e38ba93-ce7b-4613-8759-6d29cfaa2dac"). InnerVolumeSpecName "kube-api-access-pc4tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:45.700063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.700038 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca" pod "3e38ba93-ce7b-4613-8759-6d29cfaa2dac" (UID: "3e38ba93-ce7b-4613-8759-6d29cfaa2dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:45.798923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.798888 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4tx\" (UniqueName: \"kubernetes.io/projected/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-kube-api-access-pc4tx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:45.798923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.798915 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:45.799122 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:45.798931 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e38ba93-ce7b-4613-8759-6d29cfaa2dac-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.082378 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.082351 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:11:46.202887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.202839 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.202910 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.202947 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203075 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203001 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203080 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203207 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203308 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203241 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:46.203589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203266 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203351 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:46.203727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203364 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:46.203727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203621 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:46.203727 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203646 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203725 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\") pod \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\" (UID: \"423b2907-a3d9-4df1-b889-f7c725d4c9ae\") " Oct 30 20:11:46.203902 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.203862 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:46.204137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.204113 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.204232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.204142 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.204232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.204156 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.204232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.204170 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.204232 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.204183 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.205266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.205239 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:46.205370 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.205301 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "kube-api-access-5rw8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:46.205414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.205387 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:46.205701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.205684 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo" pod "423b2907-a3d9-4df1-b889-f7c725d4c9ae" (UID: "423b2907-a3d9-4df1-b889-f7c725d4c9ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:46.305155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.305127 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rw8n\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-kube-api-access-5rw8n\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.305155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.305155 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.305328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.305164 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.305328 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.305173 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b2907-a3d9-4df1-b889-f7c725d4c9ae-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:46.425620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.425583 4990 generic.go:356] "Generic (PLEG): container finished" podID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerID="ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc" exitCode=0 Oct 30 20:11:46.425786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.425645 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" Oct 30 20:11:46.425786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.425666 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerDied","Data":"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc"} Oct 30 20:11:46.425786 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.425761 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs" event={"ID":"423b2907-a3d9-4df1-b889-f7c725d4c9ae","Type":"ContainerDied","Data":"ffcdff9397c550e8266843b1f74b32d1f56676cbd84023750fd5fb8081af3379"} Oct 30 20:11:46.425969 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.425791 4990 scope.go:122] "RemoveContainer" containerID="ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc" Oct 30 20:11:46.427753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.427698 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" event={"ID":"3e38ba93-ce7b-4613-8759-6d29cfaa2dac","Type":"ContainerDied","Data":"3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107"} Oct 30 20:11:46.427855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.427767 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3022a236462d16a543c625fb65b5c609e78340d421c31c1d5f378e8aff242107" Oct 30 20:11:46.427855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.427824 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7" Oct 30 20:11:46.436116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.436077 4990 scope.go:122] "RemoveContainer" containerID="e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec" Oct 30 20:11:46.444630 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.444615 4990 scope.go:122] "RemoveContainer" containerID="ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc" Oct 30 20:11:46.444896 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:46.444874 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc\": container with ID starting with ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc not found: ID does not exist" containerID="ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc" Oct 30 20:11:46.444942 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.444905 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc"} err="failed to get container status \"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc\": rpc error: code = NotFound desc = could not find container \"ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc\": container with ID starting with ba5310e6ed63ef641add1258593472ef40fb3a8ca73c8a866f66cd1d057a89cc not found: ID does not exist" Oct 30 20:11:46.444942 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.444922 4990 scope.go:122] "RemoveContainer" containerID="e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec" Oct 30 20:11:46.445150 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:46.445136 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec\": container with ID starting with e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec not found: ID does not exist" containerID="e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec" Oct 30 20:11:46.445193 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.445153 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec"} err="failed to get container status \"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec\": rpc error: code = NotFound desc = could not find container \"e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec\": container with ID starting with e9cd31bbe251c2f7ee451300656127ebfa203948202e785fbac28513bd5b26ec not found: ID does not exist" Oct 30 20:11:46.493316 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.493290 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:11:46.497251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:46.497232 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-013647-openshift-default-765765cdb5l2ggs"] Oct 30 20:11:48.398703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:48.398671 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" path="/var/lib/kubelet/pods/423b2907-a3d9-4df1-b889-f7c725d4c9ae/volumes" Oct 30 20:11:51.872937 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:51.872903 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:52.878499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:52.878461 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:56.308113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.292526 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:11:56.378113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.369376 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\") pod \"d3af0089-ee49-4bea-85d8-5e9890425f77\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " Oct 30 20:11:56.378113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.369449 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\" (UniqueName: \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\") pod \"d3af0089-ee49-4bea-85d8-5e9890425f77\" (UID: \"d3af0089-ee49-4bea-85d8-5e9890425f77\") " Oct 30 20:11:56.383390 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.383326 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume" pod "d3af0089-ee49-4bea-85d8-5e9890425f77" (UID: "d3af0089-ee49-4bea-85d8-5e9890425f77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:56.418366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.417254 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp" pod "d3af0089-ee49-4bea-85d8-5e9890425f77" (UID: "d3af0089-ee49-4bea-85d8-5e9890425f77"). InnerVolumeSpecName "kube-api-access-fcxrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:56.458321 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.458294 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:11:56.470503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.470475 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\" (UniqueName: \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\") pod \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " Oct 30 20:11:56.470653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.470557 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\") pod \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\" (UID: \"c808c09c-783a-4a2e-b1c7-b3aa84852b9a\") " Oct 30 20:11:56.471169 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.470922 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af0089-ee49-4bea-85d8-5e9890425f77-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:56.471169 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.470946 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxrp\" (UniqueName: \"kubernetes.io/projected/d3af0089-ee49-4bea-85d8-5e9890425f77-kube-api-access-fcxrp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:56.471699 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.471674 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume" pod "c808c09c-783a-4a2e-b1c7-b3aa84852b9a" (UID: "c808c09c-783a-4a2e-b1c7-b3aa84852b9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:56.472759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.472737 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69" pod "c808c09c-783a-4a2e-b1c7-b3aa84852b9a" (UID: "c808c09c-783a-4a2e-b1c7-b3aa84852b9a"). InnerVolumeSpecName "kube-api-access-r6d69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:56.510056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.505406 4990 generic.go:356] "Generic (PLEG): container finished" podID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerID="53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f" exitCode=0 Oct 30 20:11:56.510056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.505490 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" event={"ID":"c808c09c-783a-4a2e-b1c7-b3aa84852b9a","Type":"ContainerDied","Data":"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f"} Oct 30 20:11:56.510056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.505538 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" event={"ID":"c808c09c-783a-4a2e-b1c7-b3aa84852b9a","Type":"ContainerDied","Data":"0e60fb85d6a67294ff692428b71f8694b371c6891fa275c6e8e4f397f27e5751"} Oct 30 20:11:56.510056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.505566 4990 scope.go:122] "RemoveContainer" containerID="53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f" Oct 30 20:11:56.510056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.505714 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb" Oct 30 20:11:56.513004 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.512975 4990 generic.go:356] "Generic (PLEG): container finished" podID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerID="cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea" exitCode=0 Oct 30 20:11:56.513132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.513065 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" event={"ID":"d3af0089-ee49-4bea-85d8-5e9890425f77","Type":"ContainerDied","Data":"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea"} Oct 30 20:11:56.513132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.513122 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" event={"ID":"d3af0089-ee49-4bea-85d8-5e9890425f77","Type":"ContainerDied","Data":"281eee449dc9a25a89734a6b56998b8a7b51d2531fd187cdf6df1bd46413ef0f"} Oct 30 20:11:56.513286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.513192 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg" Oct 30 20:11:56.537288 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.537224 4990 scope.go:122] "RemoveContainer" containerID="53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f" Oct 30 20:11:56.537612 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:56.537589 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f\": container with ID starting with 53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f not found: ID does not exist" containerID="53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f" Oct 30 20:11:56.537767 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.537742 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f"} err="failed to get container status \"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f\": rpc error: code = NotFound desc = could not find container \"53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f\": container with ID starting with 53bc1c91ff3d1d6bb9fe1c4847b41604848b945a6ff79cc11389d89ca632088f not found: ID does not exist" Oct 30 20:11:56.537847 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.537837 4990 scope.go:122] "RemoveContainer" containerID="cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea" Oct 30 20:11:56.547698 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.547616 4990 scope.go:122] "RemoveContainer" containerID="cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea" Oct 30 20:11:56.547909 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:11:56.547884 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea\": container with ID starting with cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea not found: ID does not exist" containerID="cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea" Oct 30 20:11:56.548030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.547911 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea"} err="failed to get container status \"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea\": rpc error: code = NotFound desc = could not find container \"cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea\": container with ID starting with cd8a0ea160c6133c588032357207094bca8581638c271faa087d89bc59a134ea not found: ID does not exist" Oct 30 20:11:56.568271 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.568229 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:11:56.571923 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.571896 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6d69\" (UniqueName: \"kubernetes.io/projected/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-kube-api-access-r6d69\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:56.572056 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.571936 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c808c09c-783a-4a2e-b1c7-b3aa84852b9a-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:11:56.572662 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.572641 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-013647-5dcfdzndcb"] Oct 30 20:11:56.602707 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.602652 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:11:56.606254 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:56.606229 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-013647-6f98d977494lzg"] Oct 30 20:11:58.399246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:58.399204 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" path="/var/lib/kubelet/pods/c808c09c-783a-4a2e-b1c7-b3aa84852b9a/volumes" Oct 30 20:11:58.399770 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:58.399750 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" path="/var/lib/kubelet/pods/d3af0089-ee49-4bea-85d8-5e9890425f77/volumes" Oct 30 20:11:58.549615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:58.549582 4990 scope.go:122] "RemoveContainer" containerID="0793ba0fbbb29278f0fa9898b126f53de2193a0d43ff3e4a564af3f3911dcea4" Oct 30 20:11:58.559653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:11:58.559633 4990 scope.go:122] "RemoveContainer" containerID="13e30e6554509a59116d472ec45a18baa1d54a2211c74dd783737c8cd1bc14f0" Oct 30 20:12:02.053383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:02.053344 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.400130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:02.400026 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.476202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:04.476143 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.095180 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:12:05.095138 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:12:06.882792 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:06.882753 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 20:12:06.883275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:06.883002 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" podUID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" containerName="manager" containerID="cri-o://e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26" gracePeriod=10 Oct 30 20:12:07.129220 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.129192 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 20:12:07.166164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.166061 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\" (UniqueName: \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\") pod \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " Oct 30 20:12:07.166289 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.166239 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\") pod \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\" (UID: \"7796eed6-f1ac-4ed3-9a48-18f44bbca748\") " Oct 30 20:12:07.166933 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.166898 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume" pod "7796eed6-f1ac-4ed3-9a48-18f44bbca748" (UID: "7796eed6-f1ac-4ed3-9a48-18f44bbca748"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:07.168355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.168330 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg" pod "7796eed6-f1ac-4ed3-9a48-18f44bbca748" (UID: "7796eed6-f1ac-4ed3-9a48-18f44bbca748"). InnerVolumeSpecName "kube-api-access-cq2fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:07.267397 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.267362 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2fg\" (UniqueName: \"kubernetes.io/projected/7796eed6-f1ac-4ed3-9a48-18f44bbca748-kube-api-access-cq2fg\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:12:07.267397 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.267390 4990 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7796eed6-f1ac-4ed3-9a48-18f44bbca748-extensions-socket-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:12:07.585700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.585661 4990 generic.go:356] "Generic (PLEG): container finished" podID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" containerID="e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26" exitCode=0 Oct 30 20:12:07.585885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.585706 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" event={"ID":"7796eed6-f1ac-4ed3-9a48-18f44bbca748","Type":"ContainerDied","Data":"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26"} Oct 30 20:12:07.585885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.585747 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" event={"ID":"7796eed6-f1ac-4ed3-9a48-18f44bbca748","Type":"ContainerDied","Data":"1b48d8c1a320d169a336bfa5121cc56f570e7b5bbb96e5de8c380081bf63168e"} Oct 30 20:12:07.585885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.585768 4990 scope.go:122] "RemoveContainer" containerID="e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26" Oct 30 20:12:07.585885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.585710 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz" Oct 30 20:12:07.595910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.595890 4990 scope.go:122] "RemoveContainer" containerID="e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26" Oct 30 20:12:07.596203 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:12:07.596174 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26\": container with ID starting with e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26 not found: ID does not exist" containerID="e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26" Oct 30 20:12:07.596281 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.596212 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26"} err="failed to get container status \"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26\": rpc error: code = NotFound desc = could not find container \"e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26\": container with ID starting with e77ab9ec12953fbcb6848a4fe818fdb802c8a8e5678846e345953789d1a3df26 not found: ID does not exist" Oct 30 20:12:07.639596 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.639570 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 20:12:07.649217 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:07.649190 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m9dxz"] Oct 30 20:12:08.125452 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:08.125415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:08.399463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:08.399392 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" path="/var/lib/kubelet/pods/7796eed6-f1ac-4ed3-9a48-18f44bbca748/volumes" Oct 30 20:12:13.028721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:13.028680 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc"] Oct 30 20:12:13.040567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:13.040539 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wvzcc"] Oct 30 20:12:14.398395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:14.398353 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8e9f49-8232-464c-8b90-9cef0723c0fe" path="/var/lib/kubelet/pods/8d8e9f49-8232-464c-8b90-9cef0723c0fe/volumes" Oct 30 20:12:23.027769 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:23.027736 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv"] Oct 30 20:12:23.039721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:23.039694 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-k2jlv"] Oct 30 20:12:24.398984 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:24.398949 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290e676c-8152-4cac-a74e-cc36dbc0125a" path="/var/lib/kubelet/pods/290e676c-8152-4cac-a74e-cc36dbc0125a/volumes" Oct 30 20:12:35.095317 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:12:35.095222 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:12:46.028154 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:46.028116 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x"] Oct 30 20:12:46.037545 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:46.037519 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d6x9x"] Oct 30 20:12:46.398477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:46.398447 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da99b164-605e-465c-893a-a747e2c8c165" path="/var/lib/kubelet/pods/da99b164-605e-465c-893a-a747e2c8c165/volumes" Oct 30 20:12:58.600295 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:58.600264 4990 scope.go:122] "RemoveContainer" containerID="f3afeb5978058c995db11add27aa2a2b99dc3bce1e14b30eaab3889a4656f097" Oct 30 20:12:58.608971 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:12:58.608951 4990 scope.go:122] "RemoveContainer" containerID="32d5b8d593536cdfbb880ec3ee8f44f523c696af01f0f926c6771f42c97d46da" Oct 30 20:13:04.603921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:04.603880 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.094983 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:13:05.094947 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:13:07.871148 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871084 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp"] Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871579 4990 cpu_manager.go:414] "containerMap: removing container" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871596 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871608 4990 cpu_manager.go:414] "containerMap: removing container" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871613 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871629 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" containerName="manager" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871636 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" containerName="manager" Oct 30 20:13:07.871651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871653 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871659 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871670 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3e38ba93-ce7b-4613-8759-6d29cfaa2dac" containerName="revoke-keys" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871676 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e38ba93-ce7b-4613-8759-6d29cfaa2dac" containerName="revoke-keys" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871741 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7796eed6-f1ac-4ed3-9a48-18f44bbca748" containerName="manager" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871751 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871758 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c808c09c-783a-4a2e-b1c7-b3aa84852b9a" containerName="payload-pre-processing" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871765 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e38ba93-ce7b-4613-8759-6d29cfaa2dac" containerName="revoke-keys" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871771 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3af0089-ee49-4bea-85d8-5e9890425f77" containerName="payload-processing" Oct 30 20:13:07.871898 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.871776 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:07.874709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.874689 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:07.898915 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.898892 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzmp6\"" Oct 30 20:13:07.901043 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:07.901025 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp"] Oct 30 20:13:08.015458 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.015425 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4p2\" (UniqueName: \"kubernetes.io/projected/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-kube-api-access-pm4p2\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.015622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.015501 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.029960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.029933 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8"] Oct 30 20:13:08.034592 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.034569 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2szs8"] Oct 30 20:13:08.116503 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.116480 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.116620 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.116531 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4p2\" (UniqueName: \"kubernetes.io/projected/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-kube-api-access-pm4p2\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.116868 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.116843 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.134243 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.134186 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4p2\" (UniqueName: \"kubernetes.io/projected/ad2b1fe0-8a6f-402b-a834-a6410a77fc72-kube-api-access-pm4p2\") pod \"kuadrant-operator-controller-manager-57bb9bfff-qpmtp\" (UID: \"ad2b1fe0-8a6f-402b-a834-a6410a77fc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.187218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.187194 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.399527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.399451 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfb09ab-b805-4bc5-92a2-4b5185de5ba9" path="/var/lib/kubelet/pods/7cfb09ab-b805-4bc5-92a2-4b5185de5ba9/volumes" Oct 30 20:13:08.514208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.514125 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp"] Oct 30 20:13:08.519780 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:13:08.519742 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2b1fe0_8a6f_402b_a834_a6410a77fc72.slice/crio-4f14d29cddba1a42ac567a44a601ef7d003a93c5fb9358197a16c6f7126b8ae5 WatchSource:0}: Error finding container 4f14d29cddba1a42ac567a44a601ef7d003a93c5fb9358197a16c6f7126b8ae5: Status 404 returned error can't find the container with id 4f14d29cddba1a42ac567a44a601ef7d003a93c5fb9358197a16c6f7126b8ae5 Oct 30 20:13:08.929440 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.929404 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" event={"ID":"ad2b1fe0-8a6f-402b-a834-a6410a77fc72","Type":"ContainerStarted","Data":"4b1d5c680a1e894d6265efd00ff35bb18f96570855d9672c4f04a7d980185736"} Oct 30 20:13:08.929957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.929462 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" event={"ID":"ad2b1fe0-8a6f-402b-a834-a6410a77fc72","Type":"ContainerStarted","Data":"4f14d29cddba1a42ac567a44a601ef7d003a93c5fb9358197a16c6f7126b8ae5"} Oct 30 20:13:08.929957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.929508 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:08.987152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:08.987078 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" podStartSLOduration=1.987062702 podStartE2EDuration="1.987062702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:08.975737739 +0000 UTC m=+4096.687263118" watchObservedRunningTime="2026-10-30 20:13:08.987062702 +0000 UTC m=+4096.698588072" Oct 30 20:13:15.268256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:15.268214 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:19.936990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:19.936958 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-qpmtp" Oct 30 20:13:29.039117 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:29.027039 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf"] Oct 30 20:13:29.043113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:29.040150 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wvhdf"] Oct 30 20:13:30.398832 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.398791 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64346a5f-7b5f-4054-9742-d77152586b6f" path="/var/lib/kubelet/pods/64346a5f-7b5f-4054-9742-d77152586b6f/volumes" Oct 30 20:13:30.680478 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.680397 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq"] Oct 30 20:13:30.680754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.680740 4990 cpu_manager.go:414] "containerMap: removing container" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:30.680803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.680755 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423b2907-a3d9-4df1-b889-f7c725d4c9ae" containerName="istio-proxy" Oct 30 20:13:30.684856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.684833 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.700309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.700285 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:30.700309 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.700287 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:30.710541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.710500 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:13:30.710687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.710554 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:30.713191 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.713168 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq"] Oct 30 20:13:30.747212 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.747182 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.747361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.747221 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.747361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.747241 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gm5\" (UniqueName: \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.847968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.847935 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.848161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.847978 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.848161 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.847998 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gm5\" (UniqueName: \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.848639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.848607 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.850381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.850357 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.864755 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.864732 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gm5\" (UniqueName: \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\") pod \"maas-api-revoke-sub-d576e68b-drvcq\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:30.997690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:30.997622 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:31.293460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:31.293385 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq"] Oct 30 20:13:31.296728 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:13:31.296701 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127fb895_909b_4724_a6b4_05aeecd3a148.slice/crio-05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33 WatchSource:0}: Error finding container 05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33: Status 404 returned error can't find the container with id 05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33 Oct 30 20:13:32.081341 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:32.081303 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" event={"ID":"127fb895-909b-4724-a6b4-05aeecd3a148","Type":"ContainerStarted","Data":"be973e35d0f52f5d47196efeb4565737e7c66f6dc6982185b97c73ab17fbbc8f"} Oct 30 20:13:32.081782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:32.081358 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" event={"ID":"127fb895-909b-4724-a6b4-05aeecd3a148","Type":"ContainerStarted","Data":"05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33"} Oct 30 20:13:32.118055 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:32.118003 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" podStartSLOduration=2.117988224 podStartE2EDuration="2.117988224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:32.115173526 +0000 UTC m=+4119.826698891" watchObservedRunningTime="2026-10-30 20:13:32.117988224 +0000 UTC m=+4119.829513589" Oct 30 20:13:33.087855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:33.087817 4990 generic.go:356] "Generic (PLEG): container finished" podID="127fb895-909b-4724-a6b4-05aeecd3a148" containerID="be973e35d0f52f5d47196efeb4565737e7c66f6dc6982185b97c73ab17fbbc8f" exitCode=0 Oct 30 20:13:33.088291 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:33.087886 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" event={"ID":"127fb895-909b-4724-a6b4-05aeecd3a148","Type":"ContainerDied","Data":"be973e35d0f52f5d47196efeb4565737e7c66f6dc6982185b97c73ab17fbbc8f"} Oct 30 20:13:34.226613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.226590 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:34.280455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.280423 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\") pod \"127fb895-909b-4724-a6b4-05aeecd3a148\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " Oct 30 20:13:34.280613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.280533 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\") pod \"127fb895-909b-4724-a6b4-05aeecd3a148\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " Oct 30 20:13:34.280684 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.280608 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\" (UniqueName: \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\") pod \"127fb895-909b-4724-a6b4-05aeecd3a148\" (UID: \"127fb895-909b-4724-a6b4-05aeecd3a148\") " Oct 30 20:13:34.280840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.280812 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca" pod "127fb895-909b-4724-a6b4-05aeecd3a148" (UID: "127fb895-909b-4724-a6b4-05aeecd3a148"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:34.280994 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.280889 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:34.282594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.282564 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca" pod "127fb895-909b-4724-a6b4-05aeecd3a148" (UID: "127fb895-909b-4724-a6b4-05aeecd3a148"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:34.282719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.282676 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5" pod "127fb895-909b-4724-a6b4-05aeecd3a148" (UID: "127fb895-909b-4724-a6b4-05aeecd3a148"). InnerVolumeSpecName "kube-api-access-m2gm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:34.381650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.381572 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127fb895-909b-4724-a6b4-05aeecd3a148-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:34.381650 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:34.381603 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gm5\" (UniqueName: \"kubernetes.io/projected/127fb895-909b-4724-a6b4-05aeecd3a148-kube-api-access-m2gm5\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:35.095680 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:13:35.095645 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:13:35.100581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:35.100557 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" Oct 30 20:13:35.100735 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:35.100586 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq" event={"ID":"127fb895-909b-4724-a6b4-05aeecd3a148","Type":"ContainerDied","Data":"05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33"} Oct 30 20:13:35.100735 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:35.100629 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05393ba666c76b9a13f55d22af967a16b2e29c3b477c50d8c384645a47782a33" Oct 30 20:13:40.266026 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:40.265991 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:47.582500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.582466 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52"] Oct 30 20:13:47.582904 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.582839 4990 cpu_manager.go:414] "containerMap: removing container" podUID="127fb895-909b-4724-a6b4-05aeecd3a148" containerName="revoke-keys" Oct 30 20:13:47.582904 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.582850 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127fb895-909b-4724-a6b4-05aeecd3a148" containerName="revoke-keys" Oct 30 20:13:47.582904 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.582905 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="127fb895-909b-4724-a6b4-05aeecd3a148" containerName="revoke-keys" Oct 30 20:13:47.586152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.586135 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.598722 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.598698 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:47.608390 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.608364 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:47.608523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.608409 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:47.608523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.608408 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:13:47.610912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.610879 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52"] Oct 30 20:13:47.726613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.726575 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h57g\" (UniqueName: \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.726613 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.726617 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.726829 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.726642 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.827496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.827454 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h57g\" (UniqueName: \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.827496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.827501 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.827769 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.827540 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.828355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.828334 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.829958 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.829933 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.845052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.844999 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h57g\" (UniqueName: \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\") pod \"maas-api-revoke-sub-d2bc93bc-nsb52\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:47.899010 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:47.898984 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:48.194853 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:48.194779 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52"] Oct 30 20:13:48.198747 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:13:48.198715 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df1d20c_3991_431b_b493_6a74ea337089.slice/crio-cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0 WatchSource:0}: Error finding container cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0: Status 404 returned error can't find the container with id cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0 Oct 30 20:13:49.175609 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:49.175575 4990 generic.go:356] "Generic (PLEG): container finished" podID="3df1d20c-3991-431b-b493-6a74ea337089" containerID="2ef931655e6e44c464f187a7ad0dceb0088b2aff7b578e4558fb58b065c5ac76" exitCode=0 Oct 30 20:13:49.176052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:49.175662 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" event={"ID":"3df1d20c-3991-431b-b493-6a74ea337089","Type":"ContainerDied","Data":"2ef931655e6e44c464f187a7ad0dceb0088b2aff7b578e4558fb58b065c5ac76"} Oct 30 20:13:49.176052 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:49.175729 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" event={"ID":"3df1d20c-3991-431b-b493-6a74ea337089","Type":"ContainerStarted","Data":"cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0"} Oct 30 20:13:50.025031 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.024998 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v"] Oct 30 20:13:50.035164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.035139 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9fd4v"] Oct 30 20:13:50.312750 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.312721 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:50.399582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.399552 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f20d754-43ad-4494-b96c-26f8901d0af1" path="/var/lib/kubelet/pods/0f20d754-43ad-4494-b96c-26f8901d0af1/volumes" Oct 30 20:13:50.455181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.455151 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\" (UniqueName: \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\") pod \"3df1d20c-3991-431b-b493-6a74ea337089\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " Oct 30 20:13:50.455343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.455188 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\") pod \"3df1d20c-3991-431b-b493-6a74ea337089\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " Oct 30 20:13:50.455343 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.455241 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\") pod \"3df1d20c-3991-431b-b493-6a74ea337089\" (UID: \"3df1d20c-3991-431b-b493-6a74ea337089\") " Oct 30 20:13:50.455642 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.455619 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca" pod "3df1d20c-3991-431b-b493-6a74ea337089" (UID: "3df1d20c-3991-431b-b493-6a74ea337089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:50.457399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.457366 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca" pod "3df1d20c-3991-431b-b493-6a74ea337089" (UID: "3df1d20c-3991-431b-b493-6a74ea337089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:50.457526 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.457424 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g" pod "3df1d20c-3991-431b-b493-6a74ea337089" (UID: "3df1d20c-3991-431b-b493-6a74ea337089"). InnerVolumeSpecName "kube-api-access-2h57g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:50.557521 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.557488 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h57g\" (UniqueName: \"kubernetes.io/projected/3df1d20c-3991-431b-b493-6a74ea337089-kube-api-access-2h57g\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:50.557676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.557525 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:50.557676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:50.557544 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df1d20c-3991-431b-b493-6a74ea337089-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:13:51.187871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:51.187838 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" Oct 30 20:13:51.188053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:51.187868 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52" event={"ID":"3df1d20c-3991-431b-b493-6a74ea337089","Type":"ContainerDied","Data":"cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0"} Oct 30 20:13:51.188053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:51.187942 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbbd5d51732ac0ab8114c9c995033a4b7eb843dde43f3b2e7ec6625a1b4747f0" Oct 30 20:13:55.989007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:55.988970 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:56.132449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:56.132409 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:58.640971 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:58.640929 4990 scope.go:122] "RemoveContainer" containerID="6db6205d131d2dcefb50e8f7e2410bb51bbb588a5b408d749ef3c426f99b6f72" Oct 30 20:13:58.650261 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:58.650237 4990 scope.go:122] "RemoveContainer" containerID="2b0a74aa6d3310c29da5559733545c5ae263630dfe2234f388903f929871a5c6" Oct 30 20:13:58.658713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:13:58.658695 4990 scope.go:122] "RemoveContainer" containerID="05b25c790ec1a7b578017d05d4bd4603f08a66bc14a612646a1fe4d4becdf8bf" Oct 30 20:14:03.732082 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:03.732039 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:04.714147 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:04.714041 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.095695 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:14:05.095654 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:14:09.387496 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.387462 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:09.764467 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.764382 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:14:09.765238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.765213 4990 cpu_manager.go:414] "containerMap: removing container" podUID="3df1d20c-3991-431b-b493-6a74ea337089" containerName="revoke-keys" Oct 30 20:14:09.765399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.765383 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df1d20c-3991-431b-b493-6a74ea337089" containerName="revoke-keys" Oct 30 20:14:09.765643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.765609 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df1d20c-3991-431b-b493-6a74ea337089" containerName="revoke-keys" Oct 30 20:14:09.769055 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.769035 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.782576 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.782552 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0bdca7-openshift-default-dockercfg-f6skb\"" Oct 30 20:14:09.795384 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.795361 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:14:09.820363 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820335 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzk8\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820396 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820457 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820572 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820652 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820697 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820716 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820736 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.820901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.820859 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.855899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.855867 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:09.921966 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.921930 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzk8\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.921979 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922014 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922033 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922053 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922074 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922118 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922147 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922175 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922487 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922510 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.922881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922858 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.923003 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.922978 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.923531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.923505 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.930134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.925180 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.930134 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.925661 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.940182 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.940139 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.940383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.940340 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzk8\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\") pod \"e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:09.997209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:09.997170 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.086012 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:10.085978 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:10.399938 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:10.399866 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:14:10.409763 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:10.409697 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:10.409968 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:10.409945 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:10.410051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:10.410007 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:11.293130 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:11.293078 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerStarted","Data":"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5"} Oct 30 20:14:11.293312 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:11.293176 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerStarted","Data":"5d4e97fff80c076553764f79b05cb9d70b55dd346fa8c6061fc9e25b5c31f3d3"} Oct 30 20:14:11.351326 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:11.351278 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podStartSLOduration=2.351263855 podStartE2EDuration="2.351263855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:11.340746921 +0000 UTC m=+4159.052272287" watchObservedRunningTime="2026-10-30 20:14:11.351263855 +0000 UTC m=+4159.062789220" Oct 30 20:14:12.086723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:12.086688 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:12.091824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:12.091798 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:12.298330 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:12.298298 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:12.299297 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:12.299277 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:22.293030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:22.292993 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:22.477662 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:22.477590 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:23.932339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:23.932293 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.143516 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.143478 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.215219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.215132 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:14:24.218831 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.218815 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.250393 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.250367 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0bdca7\"" Oct 30 20:14:24.250556 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.250396 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-69b6ce8e-dockercfg-svlxt\"" Oct 30 20:14:24.252912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.252888 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:14:24.303025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.302989 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:14:24.306619 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.306594 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.329671 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.329635 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:14:24.349128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.349046 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.359477 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.359441 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.359730 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.359710 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.359883 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.359867 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m767j\" (UniqueName: \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.360163 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.360138 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2psb\" (UniqueName: \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.461406 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.461371 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2psb\" (UniqueName: \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.461615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.461465 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.461615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.461495 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.461615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.461535 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m767j\" (UniqueName: \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.462295 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.462266 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.462417 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.462358 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.481974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.481913 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2psb\" (UniqueName: \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\") pod \"payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.482112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.482042 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m767j\" (UniqueName: \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\") pod \"payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.531407 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.531377 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:24.620592 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.620559 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:24.833704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.833669 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:14:24.837247 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:14:24.837221 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356e9aef_6437_436b_a6d0_d6633a463e7c.slice/crio-eaae69ca021f7bc187d84078131d74951dece54cca05fcf00c792f0c4d70d861 WatchSource:0}: Error finding container eaae69ca021f7bc187d84078131d74951dece54cca05fcf00c792f0c4d70d861: Status 404 returned error can't find the container with id eaae69ca021f7bc187d84078131d74951dece54cca05fcf00c792f0c4d70d861 Oct 30 20:14:24.921171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:24.921143 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:14:24.924398 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:14:24.924368 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6720ba35_88af_4e48_ab3e_3ffc005a8935.slice/crio-71d495b097d09d2aa5fe1bbb0c63ac8c3ec235b44cdf6a115da8a096982db9a9 WatchSource:0}: Error finding container 71d495b097d09d2aa5fe1bbb0c63ac8c3ec235b44cdf6a115da8a096982db9a9: Status 404 returned error can't find the container with id 71d495b097d09d2aa5fe1bbb0c63ac8c3ec235b44cdf6a115da8a096982db9a9 Oct 30 20:14:25.370958 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.370872 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" event={"ID":"6720ba35-88af-4e48-ab3e-3ffc005a8935","Type":"ContainerStarted","Data":"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359"} Oct 30 20:14:25.371480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.371058 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:25.371480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.371113 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" event={"ID":"6720ba35-88af-4e48-ab3e-3ffc005a8935","Type":"ContainerStarted","Data":"71d495b097d09d2aa5fe1bbb0c63ac8c3ec235b44cdf6a115da8a096982db9a9"} Oct 30 20:14:25.372563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.372535 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" event={"ID":"356e9aef-6437-436b-a6d0-d6633a463e7c","Type":"ContainerStarted","Data":"0c392d20f3d84a3f38a03b1f0319ba0b086a4157ebc190cc313cfd6722619246"} Oct 30 20:14:25.372711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.372578 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" event={"ID":"356e9aef-6437-436b-a6d0-d6633a463e7c","Type":"ContainerStarted","Data":"eaae69ca021f7bc187d84078131d74951dece54cca05fcf00c792f0c4d70d861"} Oct 30 20:14:25.372711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.372597 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:25.407325 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.407278 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" podStartSLOduration=1.407261601 podStartE2EDuration="1.407261601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:25.406896934 +0000 UTC m=+4173.118422300" watchObservedRunningTime="2026-10-30 20:14:25.407261601 +0000 UTC m=+4173.118786967" Oct 30 20:14:25.441862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.441815 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" podStartSLOduration=1.441801712 podStartE2EDuration="1.441801712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:25.439303264 +0000 UTC m=+4173.150828631" watchObservedRunningTime="2026-10-30 20:14:25.441801712 +0000 UTC m=+4173.153327078" Oct 30 20:14:25.702362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:25.702055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:27.352414 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:27.352368 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:28.956060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:28.956020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:30.513106 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:30.513043 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.229383 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:32.229339 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:33.947922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:33.947862 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:35.101112 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:14:35.101059 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:14:35.489318 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:35.489213 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:36.382840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:36.382812 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:14:36.383283 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:36.383073 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:14:37.221402 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.221320 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:37.447928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.447895 4990 generic.go:356] "Generic (PLEG): container finished" podID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerID="9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5" exitCode=0 Oct 30 20:14:37.448400 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.447953 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerDied","Data":"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5"} Oct 30 20:14:37.448400 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.448320 4990 scope.go:122] "RemoveContainer" containerID="9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5" Oct 30 20:14:37.449520 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.449477 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:37.449601 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.449550 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:37.449601 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:37.449581 4990 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:14:38.455189 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:38.455151 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerStarted","Data":"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d"} Oct 30 20:14:38.853402 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:38.853359 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:39.086463 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:39.086427 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:39.088219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:39.088189 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 30 20:14:39.088448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:39.088420 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 30 20:14:40.090119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:40.087454 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 30 20:14:40.090119 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:40.087533 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 30 20:14:40.504686 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:40.504595 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:41.086560 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:41.086516 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 30 20:14:41.086866 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:41.086834 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 30 20:14:42.086541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:42.086498 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 30 20:14:42.087174 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:42.087139 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 30 20:14:42.298451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:42.298401 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:42.321561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:42.321516 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:43.086564 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:43.086523 4990 patch_prober.go:29] interesting pod/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Oct 30 20:14:43.087208 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:43.087176 4990 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Oct 30 20:14:43.943563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:43.943519 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:44.101697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:44.101665 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:44.102720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:44.102697 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:14:45.501391 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:45.501355 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.190331 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:47.190294 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.391105 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:47.391055 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.804814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:47.804760 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.962928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:47.962892 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.015682 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.015642 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.077286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.077254 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:14:49.079840 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.079822 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.094303 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.094277 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0bdca7\"/\"kube-root-ca.crt\"" Oct 30 20:14:49.106438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.106410 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0bdca7\"/\"openshift-service-ca.crt\"" Oct 30 20:14:49.106561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.106451 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0bdca7\"/\"simulator-single-main-0bdca7-kserve-self-signed-certs\"" Oct 30 20:14:49.106618 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.106564 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0bdca7\"/\"default-dockercfg-q5lp9\"" Oct 30 20:14:49.108893 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.108873 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:14:49.182451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.182415 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.212518 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212484 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.212717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212531 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.212717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212562 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.212717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212701 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.212924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212740 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.212924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.212830 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pfp\" (UniqueName: \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314359 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314319 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314395 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314447 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314507 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314531 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.314607 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.314577 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pfp\" (UniqueName: \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.315352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.315325 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.315595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.315576 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.316356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.316332 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.320903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.320875 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.321393 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.321371 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.338721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.338691 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pfp\" (UniqueName: \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\") pod \"simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.391087 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.391057 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:49.691448 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:49.691420 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:14:49.693389 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:14:49.693354 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63136e02_375b_45d4_be14_6acfd86094b6.slice/crio-fe31d4f518f4c76c0ac2b63b19f5188cb9aa480e2c93974fab5a49fc30c2d2e0 WatchSource:0}: Error finding container fe31d4f518f4c76c0ac2b63b19f5188cb9aa480e2c93974fab5a49fc30c2d2e0: Status 404 returned error can't find the container with id fe31d4f518f4c76c0ac2b63b19f5188cb9aa480e2c93974fab5a49fc30c2d2e0 Oct 30 20:14:50.546641 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:50.546600 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" event={"ID":"63136e02-375b-45d4-be14-6acfd86094b6","Type":"ContainerStarted","Data":"ff98df99ae60bab19d5a3fabf853c621460b8636ae0912a02fdbb5d154623794"} Oct 30 20:14:50.546641 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:50.546648 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" event={"ID":"63136e02-375b-45d4-be14-6acfd86094b6","Type":"ContainerStarted","Data":"fe31d4f518f4c76c0ac2b63b19f5188cb9aa480e2c93974fab5a49fc30c2d2e0"} Oct 30 20:14:50.585438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:50.585388 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" podStartSLOduration=1.5853739789999999 podStartE2EDuration="1.585373979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:50.585064446 +0000 UTC m=+4198.296589815" watchObservedRunningTime="2026-10-30 20:14:50.585373979 +0000 UTC m=+4198.296899339" Oct 30 20:14:50.830567 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:50.830159 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:51.392133 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:51.392086 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:52.464399 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:52.464357 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.202275 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:54.202239 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.800701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:54.800668 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:14:54.810019 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:54.809994 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:14:54.827431 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:54.827408 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:14:54.836519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:54.836496 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:14:55.871813 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:55.871780 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.410256 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:57.410212 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:58.689764 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:58.689729 4990 scope.go:122] "RemoveContainer" containerID="793076f37db1baf6a404eb6263537ed748f676858e347af5d1377e68f982c29e" Oct 30 20:14:58.698720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:58.698699 4990 scope.go:122] "RemoveContainer" containerID="2e317035f6c97eacc38b999e2430575f6b50a51f4a7be011aa8ac2c869f98628" Oct 30 20:14:58.706843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:58.706819 4990 scope.go:122] "RemoveContainer" containerID="53b39fd6cce423a19e03e8be480a125acea7d90ab95977622f7158711f5905f9" Oct 30 20:14:58.714811 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:58.714791 4990 scope.go:122] "RemoveContainer" containerID="37d392e7602a00eb3715112626c0a55a4c97acf53a4a324c652301b967a0d9cd" Oct 30 20:14:58.951481 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:58.951385 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:59.391274 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:59.391238 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:59.403859 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:59.403831 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:14:59.608570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:14:59.608538 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:15:00.139978 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.139948 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:00.143901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.143879 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.171386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.171335 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:15:00.171386 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.171355 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:00.171575 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.171385 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:00.173890 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.173871 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:00.222180 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.222138 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.222374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.222189 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzfx\" (UniqueName: \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.323108 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.323062 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.323302 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.323127 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzfx\" (UniqueName: \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.323759 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.323732 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.341814 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.341789 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzfx\" (UniqueName: \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\") pod \"maas-api-key-cleanup-29889855-2cxz7\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.456307 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.456224 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:00.532640 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.532606 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.753002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:00.752972 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:00.755663 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:15:00.755634 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc800b569_4f97_4160_849d_523d4d26b191.slice/crio-99c0965ed464861c5ccd04ef10e5b0a94f817f489d33d40be11cefe4bb4be8ae WatchSource:0}: Error finding container 99c0965ed464861c5ccd04ef10e5b0a94f817f489d33d40be11cefe4bb4be8ae: Status 404 returned error can't find the container with id 99c0965ed464861c5ccd04ef10e5b0a94f817f489d33d40be11cefe4bb4be8ae Oct 30 20:15:01.610204 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:01.610174 4990 generic.go:356] "Generic (PLEG): container finished" podID="c800b569-4f97-4160-849d-523d4d26b191" containerID="c7bdc7f9e4551fbb17e721595297d7833a3de06cc9d97e89f3f307c4e839b5cc" exitCode=60 Oct 30 20:15:01.610589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:01.610258 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerDied","Data":"c7bdc7f9e4551fbb17e721595297d7833a3de06cc9d97e89f3f307c4e839b5cc"} Oct 30 20:15:01.610589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:01.610313 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerStarted","Data":"99c0965ed464861c5ccd04ef10e5b0a94f817f489d33d40be11cefe4bb4be8ae"} Oct 30 20:15:01.610589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:01.610485 4990 scope.go:122] "RemoveContainer" containerID="c7bdc7f9e4551fbb17e721595297d7833a3de06cc9d97e89f3f307c4e839b5cc" Oct 30 20:15:02.230332 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:02.230291 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:02.617402 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:02.617367 4990 generic.go:356] "Generic (PLEG): container finished" podID="c800b569-4f97-4160-849d-523d4d26b191" containerID="4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b" exitCode=60 Oct 30 20:15:02.617878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:02.617421 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerDied","Data":"4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b"} Oct 30 20:15:02.617878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:02.617512 4990 scope.go:122] "RemoveContainer" containerID="c7bdc7f9e4551fbb17e721595297d7833a3de06cc9d97e89f3f307c4e839b5cc" Oct 30 20:15:02.617878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:02.617848 4990 scope.go:122] "RemoveContainer" containerID="4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b" Oct 30 20:15:02.618178 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:15:02.618150 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2cxz7_odh-ai-gateway-infra(c800b569-4f97-4160-849d-523d4d26b191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" podUID="c800b569-4f97-4160-849d-523d4d26b191" Oct 30 20:15:03.624770 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:03.624741 4990 scope.go:122] "RemoveContainer" containerID="4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b" Oct 30 20:15:03.625224 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:15:03.624985 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2cxz7_odh-ai-gateway-infra(c800b569-4f97-4160-849d-523d4d26b191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" podUID="c800b569-4f97-4160-849d-523d4d26b191" Oct 30 20:15:03.810073 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:03.810030 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.533068 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:04.533022 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.833529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:04.833498 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:05.095397 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:15:05.095320 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:15:05.445913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:05.445805 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.025382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:07.025328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.761822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:08.761784 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.431981 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:10.431946 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:12.165823 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:12.165759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.712850 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:13.712810 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.394244 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:15.394211 4990 scope.go:122] "RemoveContainer" containerID="4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b" Oct 30 20:15:15.536523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:15.536482 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.691813 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:15.691782 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerStarted","Data":"25f9f19aedd5d0c278a34d8840ef9adb8f7220a9d424867781f31e0098336e88"} Oct 30 20:15:15.726546 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:15.726344 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" podStartSLOduration=15.726328877 podStartE2EDuration="15.726328877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:15.724847555 +0000 UTC m=+4223.436372923" watchObservedRunningTime="2026-10-30 20:15:15.726328877 +0000 UTC m=+4223.437854242" Oct 30 20:15:16.699710 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:16.699675 4990 generic.go:356] "Generic (PLEG): container finished" podID="c800b569-4f97-4160-849d-523d4d26b191" containerID="25f9f19aedd5d0c278a34d8840ef9adb8f7220a9d424867781f31e0098336e88" exitCode=60 Oct 30 20:15:16.700236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:16.699730 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerDied","Data":"25f9f19aedd5d0c278a34d8840ef9adb8f7220a9d424867781f31e0098336e88"} Oct 30 20:15:16.700236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:16.699790 4990 scope.go:122] "RemoveContainer" containerID="4afdc56f31053812b1ccc1a1d3ac2fcf971b31fcbb787b8caa9f2fc5f1c9e57b" Oct 30 20:15:16.700236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:16.700163 4990 scope.go:122] "RemoveContainer" containerID="25f9f19aedd5d0c278a34d8840ef9adb8f7220a9d424867781f31e0098336e88" Oct 30 20:15:16.700490 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:15:16.700458 4990 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2cxz7_odh-ai-gateway-infra(c800b569-4f97-4160-849d-523d4d26b191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" podUID="c800b569-4f97-4160-849d-523d4d26b191" Oct 30 20:15:16.741350 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:16.741321 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:17.581849 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.581811 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:17.841171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.841119 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:17.892713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.892686 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\") pod \"c800b569-4f97-4160-849d-523d4d26b191\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " Oct 30 20:15:17.892879 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.892767 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\" (UniqueName: \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\") pod \"c800b569-4f97-4160-849d-523d4d26b191\" (UID: \"c800b569-4f97-4160-849d-523d4d26b191\") " Oct 30 20:15:17.893044 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.893020 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca" pod "c800b569-4f97-4160-849d-523d4d26b191" (UID: "c800b569-4f97-4160-849d-523d4d26b191"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:17.894976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.894945 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx" pod "c800b569-4f97-4160-849d-523d4d26b191" (UID: "c800b569-4f97-4160-849d-523d4d26b191"). InnerVolumeSpecName "kube-api-access-2rzfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:17.994476 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.994437 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzfx\" (UniqueName: \"kubernetes.io/projected/c800b569-4f97-4160-849d-523d4d26b191-kube-api-access-2rzfx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:17.994476 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:17.994471 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c800b569-4f97-4160-849d-523d4d26b191-maas-api-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.714776 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.714686 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" Oct 30 20:15:18.714776 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.714704 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7" event={"ID":"c800b569-4f97-4160-849d-523d4d26b191","Type":"ContainerDied","Data":"99c0965ed464861c5ccd04ef10e5b0a94f817f489d33d40be11cefe4bb4be8ae"} Oct 30 20:15:18.714995 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.714815 4990 scope.go:122] "RemoveContainer" containerID="25f9f19aedd5d0c278a34d8840ef9adb8f7220a9d424867781f31e0098336e88" Oct 30 20:15:18.759355 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.759329 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:18.769304 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.769277 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2cxz7"] Oct 30 20:15:18.937789 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:18.937758 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.401178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:20.401137 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c800b569-4f97-4160-849d-523d4d26b191" path="/var/lib/kubelet/pods/c800b569-4f97-4160-849d-523d4d26b191/volumes" Oct 30 20:15:20.583036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:20.582993 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.199715 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:22.199649 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:23.839866 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:23.839820 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.389116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:25.389058 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:27.209438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:27.209396 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:28.751069 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.751026 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:28.948439 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948403 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4"] Oct 30 20:15:28.948849 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948826 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.948849 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948842 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948857 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948863 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948870 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948877 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948954 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948964 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.949013 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.948970 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c800b569-4f97-4160-849d-523d4d26b191" containerName="cleanup" Oct 30 20:15:28.953247 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.953230 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:28.971602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.971578 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4"] Oct 30 20:15:28.981132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.981081 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:28.981265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.981134 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:28.981265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.981165 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:28.981265 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:28.981196 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:15:29.099505 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.099463 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxvf\" (UniqueName: \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.099705 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.099515 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.099705 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.099593 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.200535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.200500 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxvf\" (UniqueName: \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.200718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.200548 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.200718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.200572 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.201260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.201232 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.202826 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.202800 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.219647 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.219622 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxvf\" (UniqueName: \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\") pod \"maas-api-revoke-sub-6a426a1b-rhrc4\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.265993 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.265966 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:29.567085 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.567051 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4"] Oct 30 20:15:29.568803 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:15:29.568775 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6a7fcf_c27f_468f_ac9f_b09dff40f1b5.slice/crio-cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8 WatchSource:0}: Error finding container cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8: Status 404 returned error can't find the container with id cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8 Oct 30 20:15:29.782435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:29.782395 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" event={"ID":"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5","Type":"ContainerStarted","Data":"cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8"} Oct 30 20:15:30.478433 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:30.478379 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.790136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:30.790018 4990 generic.go:356] "Generic (PLEG): container finished" podID="4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" containerID="99f32c97ffe5dbd891720204258c6b54b5c6f708f5f976268c7b9fc3afda9220" exitCode=0 Oct 30 20:15:30.790136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:30.790121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" event={"ID":"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5","Type":"ContainerDied","Data":"99f32c97ffe5dbd891720204258c6b54b5c6f708f5f976268c7b9fc3afda9220"} Oct 30 20:15:31.922199 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:31.922175 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:32.030042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.030011 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\") pod \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " Oct 30 20:15:32.030257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.030063 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\" (UniqueName: \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\") pod \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " Oct 30 20:15:32.030257 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.030101 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\") pod \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\" (UID: \"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5\") " Oct 30 20:15:32.030455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.030428 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca" pod "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" (UID: "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:32.032294 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.032268 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf" pod "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" (UID: "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5"). InnerVolumeSpecName "kube-api-access-7lxvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:32.032678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.032656 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca" pod "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" (UID: "4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:32.123473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.123433 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.131533 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.131499 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:32.131533 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.131530 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxvf\" (UniqueName: \"kubernetes.io/projected/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-kube-api-access-7lxvf\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:32.131533 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.131540 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:32.802592 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.802562 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" Oct 30 20:15:32.802760 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.802594 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4" event={"ID":"4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5","Type":"ContainerDied","Data":"cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8"} Oct 30 20:15:32.802760 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:32.802638 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5be7c81f7f8f92fbacb1403a350a4b451bd98517fa384740fee1eb16c974a8" Oct 30 20:15:33.740909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:33.740871 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:35.094814 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:15:35.094739 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:15:35.296913 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:35.296874 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:36.945704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:36.945673 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.623713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:38.623673 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.768737 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:38.768696 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.369327 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.369289 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.842270 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.842214 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.972176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.972143 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs"] Oct 30 20:15:39.972528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.972517 4990 cpu_manager.go:414] "containerMap: removing container" podUID="4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" containerName="revoke-keys" Oct 30 20:15:39.972578 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.972530 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" containerName="revoke-keys" Oct 30 20:15:39.972628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.972619 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" containerName="revoke-keys" Oct 30 20:15:39.975817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.975799 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:39.996164 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:39.996135 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs"] Oct 30 20:15:40.003563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.003529 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:40.003690 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.003669 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:40.003755 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.003688 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:40.003755 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.003694 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:15:40.009470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.009440 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.009661 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.009640 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.009725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.009685 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqppj\" (UniqueName: \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.111457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.111376 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.111612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.111455 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqppj\" (UniqueName: \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.111612 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.111503 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.112203 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.112173 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.113839 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.113808 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.131335 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.131313 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqppj\" (UniqueName: \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.289286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.289254 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:40.360054 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.360024 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.597142 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.597059 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs"] Oct 30 20:15:40.599729 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:15:40.599701 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5c78fb_d3e2_47c8_9454_0283a6a88150.slice/crio-4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d WatchSource:0}: Error finding container 4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d: Status 404 returned error can't find the container with id 4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d Oct 30 20:15:40.847617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:40.847535 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" event={"ID":"0d5c78fb-d3e2-47c8-9454-0283a6a88150","Type":"ContainerStarted","Data":"4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d"} Oct 30 20:15:41.855697 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:41.855659 4990 generic.go:356] "Generic (PLEG): container finished" podID="0d5c78fb-d3e2-47c8-9454-0283a6a88150" containerID="6de1d3251d6a44ae3dcc4e6f1c4b85fc9acf3f6a3b6c43dac6598174b4447cf7" exitCode=0 Oct 30 20:15:41.856190 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:41.855699 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" event={"ID":"0d5c78fb-d3e2-47c8-9454-0283a6a88150","Type":"ContainerDied","Data":"6de1d3251d6a44ae3dcc4e6f1c4b85fc9acf3f6a3b6c43dac6598174b4447cf7"} Oct 30 20:15:42.037798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:42.037759 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.001948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.001926 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:43.041141 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.041081 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\" (UniqueName: \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\") pod \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " Oct 30 20:15:43.041416 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.041389 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\") pod \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " Oct 30 20:15:43.041547 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.041460 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\") pod \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\" (UID: \"0d5c78fb-d3e2-47c8-9454-0283a6a88150\") " Oct 30 20:15:43.042852 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.042807 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca" pod "0d5c78fb-d3e2-47c8-9454-0283a6a88150" (UID: "0d5c78fb-d3e2-47c8-9454-0283a6a88150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:43.044126 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.044056 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca" pod "0d5c78fb-d3e2-47c8-9454-0283a6a88150" (UID: "0d5c78fb-d3e2-47c8-9454-0283a6a88150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:43.044227 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.044216 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj" pod "0d5c78fb-d3e2-47c8-9454-0283a6a88150" (UID: "0d5c78fb-d3e2-47c8-9454-0283a6a88150"). InnerVolumeSpecName "kube-api-access-xqppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:43.142912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.142823 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqppj\" (UniqueName: \"kubernetes.io/projected/0d5c78fb-d3e2-47c8-9454-0283a6a88150-kube-api-access-xqppj\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:43.142912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.142856 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:43.142912 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.142866 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c78fb-d3e2-47c8-9454-0283a6a88150-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:15:43.696535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.696500 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.876112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.869416 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" event={"ID":"0d5c78fb-d3e2-47c8-9454-0283a6a88150","Type":"ContainerDied","Data":"4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d"} Oct 30 20:15:43.876112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.869543 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c42cea108b71c1492e9463bffa45ba362ce12c9e2aa2b43837697b7f5b8ca0d" Oct 30 20:15:43.876112 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:43.869622 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs" Oct 30 20:15:44.759891 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:44.759855 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:15:44.880744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:44.880713 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.330251 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:45.330212 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.054885 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:47.054851 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.627155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:48.627115 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.323181 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:50.323144 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.509182 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:50.509146 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:15:50.561218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:50.561186 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:15:50.713398 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:50.713306 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.857425 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:50.857388 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.040729 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:52.040664 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.535224 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:53.535174 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.131170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.131131 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.236178 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.236086 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs"] Oct 30 20:15:54.247456 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.247420 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0bdca7-vvvvs"] Oct 30 20:15:54.296699 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.296654 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.402594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.402516 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5c78fb-d3e2-47c8-9454-0283a6a88150" path="/var/lib/kubelet/pods/0d5c78fb-d3e2-47c8-9454-0283a6a88150/volumes" Oct 30 20:15:54.828788 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:54.828744 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:55.510009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:55.509968 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" containerID="cri-o://8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359" gracePeriod=25 Oct 30 20:15:55.905803 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:55.905743 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" containerID="cri-o://0c392d20f3d84a3f38a03b1f0319ba0b086a4157ebc190cc313cfd6722619246" gracePeriod=25 Oct 30 20:15:56.381523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:56.381471 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:56.381719 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:56.381527 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:58.746028 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:58.745987 4990 scope.go:122] "RemoveContainer" containerID="0bac6ebbad7b4aa4676bc739c8af65f86fe31fae425bb79d8fae1563e29b1682" Oct 30 20:15:58.754910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:58.754887 4990 scope.go:122] "RemoveContainer" containerID="7f5ae8ceb116e43b4d5b79d48a228226b7400fad9fd54ba5951f690e437d7abc" Oct 30 20:15:59.779807 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:59.779766 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" podUID="63136e02-375b-45d4-be14-6acfd86094b6" containerName="main" containerID="cri-o://ff98df99ae60bab19d5a3fabf853c621460b8636ae0912a02fdbb5d154623794" gracePeriod=45 Oct 30 20:15:59.956214 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:59.956180 4990 generic.go:356] "Generic (PLEG): container finished" podID="63136e02-375b-45d4-be14-6acfd86094b6" containerID="ff98df99ae60bab19d5a3fabf853c621460b8636ae0912a02fdbb5d154623794" exitCode=0 Oct 30 20:15:59.956351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:15:59.956334 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" event={"ID":"63136e02-375b-45d4-be14-6acfd86094b6","Type":"ContainerDied","Data":"ff98df99ae60bab19d5a3fabf853c621460b8636ae0912a02fdbb5d154623794"} Oct 30 20:16:00.032773 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.032715 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:16:00.108639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108601 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.108834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108673 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.108834 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108779 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\" (UniqueName: \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.108925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108864 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.108925 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108903 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.109047 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.108934 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\") pod \"63136e02-375b-45d4-be14-6acfd86094b6\" (UID: \"63136e02-375b-45d4-be14-6acfd86094b6\") " Oct 30 20:16:00.109451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.109183 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:00.109451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.109209 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:00.109451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.109399 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.109451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.109408 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:00.109451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.109420 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.111473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.111446 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:00.111581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.111560 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:00.111628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.111592 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp" pod "63136e02-375b-45d4-be14-6acfd86094b6" (UID: "63136e02-375b-45d4-be14-6acfd86094b6"). InnerVolumeSpecName "kube-api-access-t2pfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:00.210266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.210229 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63136e02-375b-45d4-be14-6acfd86094b6-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.210266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.210256 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.210266 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.210267 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pfp\" (UniqueName: \"kubernetes.io/projected/63136e02-375b-45d4-be14-6acfd86094b6-kube-api-access-t2pfp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.210512 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.210276 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63136e02-375b-45d4-be14-6acfd86094b6-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:00.718615 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.718528 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:00.868385 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.868341 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:00.963502 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.963458 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" event={"ID":"63136e02-375b-45d4-be14-6acfd86094b6","Type":"ContainerDied","Data":"fe31d4f518f4c76c0ac2b63b19f5188cb9aa480e2c93974fab5a49fc30c2d2e0"} Oct 30 20:16:00.963502 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.963482 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf" Oct 30 20:16:00.963502 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:00.963506 4990 scope.go:122] "RemoveContainer" containerID="ff98df99ae60bab19d5a3fabf853c621460b8636ae0912a02fdbb5d154623794" Oct 30 20:16:01.016042 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:01.016010 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:16:01.016277 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:01.016252 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" containerID="cri-o://51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d" gracePeriod=30 Oct 30 20:16:01.019400 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:01.019370 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:16:01.028179 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:01.028150 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0bdca7/simulator-single-main-0bdca7-kserve-d9578d9df-xhxkf"] Oct 30 20:16:01.105081 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:01.105045 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.399422 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:02.399384 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63136e02-375b-45d4-be14-6acfd86094b6" path="/var/lib/kubelet/pods/63136e02-375b-45d4-be14-6acfd86094b6/volumes" Oct 30 20:16:04.961793 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:04.961754 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:05.096359 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:16:05.096320 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:16:05.187693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:05.187652 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.268182 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.268158 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:16:06.368401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368373 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368408 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368432 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368461 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368510 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368594 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368555 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368600 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368638 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368724 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\") pod \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\" (UID: \"9eb54619-4166-4dc8-ba54-c4eef71bc2ca\") " Oct 30 20:16:06.368907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368753 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:06.369139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368959 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:06.369139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.368971 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:06.369259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369162 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:06.369259 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369174 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:06.369374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369302 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.369374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369325 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.369374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369340 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.369374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369353 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.369374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.369368 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.371311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.371287 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:06.371311 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.371298 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "kube-api-access-wlzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:06.371457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.371317 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:06.371457 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.371330 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token" pod "9eb54619-4166-4dc8-ba54-c4eef71bc2ca" (UID: "9eb54619-4166-4dc8-ba54-c4eef71bc2ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:06.381471 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.381446 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:06.381581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.381497 4990 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:06.470639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.470613 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.470639 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.470638 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzk8\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-kube-api-access-wlzk8\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.470798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.470648 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:06.470798 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:06.470658 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb54619-4166-4dc8-ba54-c4eef71bc2ca-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:07.003084 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.003045 4990 generic.go:356] "Generic (PLEG): container finished" podID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerID="51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d" exitCode=0 Oct 30 20:16:07.003292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.003127 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" Oct 30 20:16:07.003292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.003144 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerDied","Data":"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d"} Oct 30 20:16:07.003292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.003243 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t" event={"ID":"9eb54619-4166-4dc8-ba54-c4eef71bc2ca","Type":"ContainerDied","Data":"5d4e97fff80c076553764f79b05cb9d70b55dd346fa8c6061fc9e25b5c31f3d3"} Oct 30 20:16:07.003292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.003274 4990 scope.go:122] "RemoveContainer" containerID="51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d" Oct 30 20:16:07.012436 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.012419 4990 scope.go:122] "RemoveContainer" containerID="9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5" Oct 30 20:16:07.020438 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.020419 4990 scope.go:122] "RemoveContainer" containerID="51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d" Oct 30 20:16:07.020695 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:16:07.020672 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d\": container with ID starting with 51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d not found: ID does not exist" containerID="51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d" Oct 30 20:16:07.020806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.020700 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d"} err="failed to get container status \"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d\": rpc error: code = NotFound desc = could not find container \"51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d\": container with ID starting with 51c255df74e2d2f3af018f1f9713f66be69618b9888e24a71c0656ebdf30cc6d not found: ID does not exist" Oct 30 20:16:07.020806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.020717 4990 scope.go:122] "RemoveContainer" containerID="9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5" Oct 30 20:16:07.020986 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:16:07.020970 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5\": container with ID starting with 9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5 not found: ID does not exist" containerID="9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5" Oct 30 20:16:07.021030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.020991 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5"} err="failed to get container status \"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5\": rpc error: code = NotFound desc = could not find container \"9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5\": container with ID starting with 9e17c726504b2e9c140a8554ebb0a42223c9c3730f8fb810c5f1e8395b4eaed5 not found: ID does not exist" Oct 30 20:16:07.066744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.066715 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:16:07.078956 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:07.078930 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0bdca7-openshift-default-7b6c7b54b59nz6t"] Oct 30 20:16:08.399470 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:08.399435 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" path="/var/lib/kubelet/pods/9eb54619-4166-4dc8-ba54-c4eef71bc2ca/volumes" Oct 30 20:16:13.032326 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:13.032294 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq"] Oct 30 20:16:13.044271 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:13.044240 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c748c1-j22pq"] Oct 30 20:16:14.399617 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:14.399584 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f926b88b-ecae-4291-9803-4635c70dec74" path="/var/lib/kubelet/pods/f926b88b-ecae-4291-9803-4635c70dec74/volumes" Oct 30 20:16:15.750590 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.750561 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:16:15.860609 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.860574 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\" (UniqueName: \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\") pod \"6720ba35-88af-4e48-ab3e-3ffc005a8935\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " Oct 30 20:16:15.860773 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.860659 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\") pod \"6720ba35-88af-4e48-ab3e-3ffc005a8935\" (UID: \"6720ba35-88af-4e48-ab3e-3ffc005a8935\") " Oct 30 20:16:15.861063 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.861032 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume" pod "6720ba35-88af-4e48-ab3e-3ffc005a8935" (UID: "6720ba35-88af-4e48-ab3e-3ffc005a8935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:15.862855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.862826 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb" pod "6720ba35-88af-4e48-ab3e-3ffc005a8935" (UID: "6720ba35-88af-4e48-ab3e-3ffc005a8935"). InnerVolumeSpecName "kube-api-access-l2psb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:15.961725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.961685 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6720ba35-88af-4e48-ab3e-3ffc005a8935-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:15.961725 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:15.961720 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2psb\" (UniqueName: \"kubernetes.io/projected/6720ba35-88af-4e48-ab3e-3ffc005a8935-kube-api-access-l2psb\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:16.052799 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.052752 4990 generic.go:356] "Generic (PLEG): container finished" podID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerID="8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359" exitCode=0 Oct 30 20:16:16.052987 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.052962 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" event={"ID":"6720ba35-88af-4e48-ab3e-3ffc005a8935","Type":"ContainerDied","Data":"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359"} Oct 30 20:16:16.053053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.053040 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" event={"ID":"6720ba35-88af-4e48-ab3e-3ffc005a8935","Type":"ContainerDied","Data":"71d495b097d09d2aa5fe1bbb0c63ac8c3ec235b44cdf6a115da8a096982db9a9"} Oct 30 20:16:16.053114 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.053067 4990 scope.go:122] "RemoveContainer" containerID="8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359" Oct 30 20:16:16.053216 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.053204 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm" Oct 30 20:16:16.058356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.058321 4990 generic.go:356] "Generic (PLEG): container finished" podID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerID="0c392d20f3d84a3f38a03b1f0319ba0b086a4157ebc190cc313cfd6722619246" exitCode=0 Oct 30 20:16:16.058486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.058393 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" event={"ID":"356e9aef-6437-436b-a6d0-d6633a463e7c","Type":"ContainerDied","Data":"0c392d20f3d84a3f38a03b1f0319ba0b086a4157ebc190cc313cfd6722619246"} Oct 30 20:16:16.070602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.070575 4990 scope.go:122] "RemoveContainer" containerID="8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359" Oct 30 20:16:16.071438 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:16:16.071410 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359\": container with ID starting with 8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359 not found: ID does not exist" containerID="8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359" Oct 30 20:16:16.071561 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.071452 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359"} err="failed to get container status \"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359\": rpc error: code = NotFound desc = could not find container \"8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359\": container with ID starting with 8417888ea018c3728f8f18aab6d55178f4c1961f8771ce44c25ef037f1ae9359 not found: ID does not exist" Oct 30 20:16:16.107871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.107829 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:16:16.119774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.119708 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0bdca7-5f77f89f4665hm"] Oct 30 20:16:16.144903 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.144881 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:16:16.264364 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.264322 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\" (UniqueName: \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\") pod \"356e9aef-6437-436b-a6d0-d6633a463e7c\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " Oct 30 20:16:16.264550 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.264427 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\") pod \"356e9aef-6437-436b-a6d0-d6633a463e7c\" (UID: \"356e9aef-6437-436b-a6d0-d6633a463e7c\") " Oct 30 20:16:16.264855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.264832 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume" pod "356e9aef-6437-436b-a6d0-d6633a463e7c" (UID: "356e9aef-6437-436b-a6d0-d6633a463e7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:16.266439 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.266413 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j" pod "356e9aef-6437-436b-a6d0-d6633a463e7c" (UID: "356e9aef-6437-436b-a6d0-d6633a463e7c"). InnerVolumeSpecName "kube-api-access-m767j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:16.365404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.365361 4990 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356e9aef-6437-436b-a6d0-d6633a463e7c-plugins-config-volume\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:16.365404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.365392 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m767j\" (UniqueName: \"kubernetes.io/projected/356e9aef-6437-436b-a6d0-d6633a463e7c-kube-api-access-m767j\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:16:16.398771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:16.398702 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" path="/var/lib/kubelet/pods/6720ba35-88af-4e48-ab3e-3ffc005a8935/volumes" Oct 30 20:16:17.073319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:17.073289 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" Oct 30 20:16:17.073851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:17.073278 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh" event={"ID":"356e9aef-6437-436b-a6d0-d6633a463e7c","Type":"ContainerDied","Data":"eaae69ca021f7bc187d84078131d74951dece54cca05fcf00c792f0c4d70d861"} Oct 30 20:16:17.073851 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:17.073489 4990 scope.go:122] "RemoveContainer" containerID="0c392d20f3d84a3f38a03b1f0319ba0b086a4157ebc190cc313cfd6722619246" Oct 30 20:16:17.123926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:17.123890 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:16:17.136140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:17.136110 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0bdca7-8489bfcfzh"] Oct 30 20:16:18.398548 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:18.398513 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" path="/var/lib/kubelet/pods/356e9aef-6437-436b-a6d0-d6633a463e7c/volumes" Oct 30 20:16:22.034403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:22.034363 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:22.276800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:22.276760 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:27.929168 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:27.929128 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:28.949460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.949421 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:16:28.949973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.949933 4990 cpu_manager.go:414] "containerMap: removing container" podUID="63136e02-375b-45d4-be14-6acfd86094b6" containerName="main" Oct 30 20:16:28.949973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.949953 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63136e02-375b-45d4-be14-6acfd86094b6" containerName="main" Oct 30 20:16:28.949973 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.949968 4990 cpu_manager.go:414] "containerMap: removing container" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.949978 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950013 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950023 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950037 4990 cpu_manager.go:414] "containerMap: removing container" podUID="0d5c78fb-d3e2-47c8-9454-0283a6a88150" containerName="revoke-keys" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950045 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5c78fb-d3e2-47c8-9454-0283a6a88150" containerName="revoke-keys" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950060 4990 cpu_manager.go:414] "containerMap: removing container" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" Oct 30 20:16:28.950177 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950068 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950188 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="356e9aef-6437-436b-a6d0-d6633a463e7c" containerName="payload-pre-processing" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950202 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950216 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="63136e02-375b-45d4-be14-6acfd86094b6" containerName="main" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950226 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="6720ba35-88af-4e48-ab3e-3ffc005a8935" containerName="payload-processing" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950240 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:28.950536 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.950249 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5c78fb-d3e2-47c8-9454-0283a6a88150" containerName="revoke-keys" Oct 30 20:16:28.955165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.955085 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.980510 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.980482 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-41fd0cc3-kserve-self-signed-certs\"" Oct 30 20:16:28.985585 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.985561 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:16:28.985777 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.985749 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.985889 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.985872 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.985991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.985909 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfsp\" (UniqueName: \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.986078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.986053 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.986236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.986109 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:28.986236 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:28.986200 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087452 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087486 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087499 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfsp\" (UniqueName: \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087580 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087606 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087643 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.087782 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.087700 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.088070 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.088050 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.088198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.088176 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.088198 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.088190 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.089868 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.089836 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.089983 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.089965 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.102532 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.102437 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:29.104693 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.104667 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfsp\" (UniqueName: \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\") pod \"e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.267901 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.267808 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:29.563514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:29.563482 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:16:29.567499 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:16:29.566902 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc57445e7_af71_4aff_a867_82c510c9261e.slice/crio-d1a6a26e77fed93d4aa42416014f61e1738882f4da17120f58ced5e0888b89eb WatchSource:0}: Error finding container d1a6a26e77fed93d4aa42416014f61e1738882f4da17120f58ced5e0888b89eb: Status 404 returned error can't find the container with id d1a6a26e77fed93d4aa42416014f61e1738882f4da17120f58ced5e0888b89eb Oct 30 20:16:30.154509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:30.154476 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" event={"ID":"c57445e7-af71-4aff-a867-82c510c9261e","Type":"ContainerStarted","Data":"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0"} Oct 30 20:16:30.154921 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:30.154521 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" event={"ID":"c57445e7-af71-4aff-a867-82c510c9261e","Type":"ContainerStarted","Data":"d1a6a26e77fed93d4aa42416014f61e1738882f4da17120f58ced5e0888b89eb"} Oct 30 20:16:30.189135 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:30.189069 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" podStartSLOduration=2.189054298 podStartE2EDuration="2.189054298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:30.188038984 +0000 UTC m=+4297.899564350" watchObservedRunningTime="2026-10-30 20:16:30.189054298 +0000 UTC m=+4297.900579658" Oct 30 20:16:30.268366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:30.268328 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:35.096137 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:16:35.096086 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:16:39.268387 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:39.268348 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:39.280523 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:39.280496 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:40.226490 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:40.226455 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:16:46.026602 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:46.026559 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7"] Oct 30 20:16:46.036058 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:46.036030 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l8dq7"] Oct 30 20:16:46.399385 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:46.399350 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e38ba93-ce7b-4613-8759-6d29cfaa2dac" path="/var/lib/kubelet/pods/3e38ba93-ce7b-4613-8759-6d29cfaa2dac/volumes" Oct 30 20:16:57.531926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.531889 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:16:57.532511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.532281 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:57.532511 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.532294 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb54619-4166-4dc8-ba54-c4eef71bc2ca" containerName="istio-proxy" Oct 30 20:16:57.534472 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.534454 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.549721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.549684 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-41fd0cc3-kserve-self-signed-certs\"" Oct 30 20:16:57.562085 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.562056 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:16:57.666997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.666955 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:57.673153 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673122 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.673296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673239 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.673296 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673285 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9pk\" (UniqueName: \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.673424 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673316 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.673424 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673338 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.673535 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.673456 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774018 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.773975 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774040 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9pk\" (UniqueName: \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774080 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774131 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774164 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774193 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774502 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774540 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774502 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.774661 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.774616 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.776605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.776579 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.776789 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.776770 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.790865 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.790806 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9pk\" (UniqueName: \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\") pod \"e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:57.846988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:57.846941 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:16:58.147121 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:58.147066 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:16:58.151384 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:16:58.151352 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab4f322_4369_489b_abdf_82b2530a0b66.slice/crio-d7aa10f639853fee6ac2674ea06e2bd934619dbe9f92b6db5dd6552b10716065 WatchSource:0}: Error finding container d7aa10f639853fee6ac2674ea06e2bd934619dbe9f92b6db5dd6552b10716065: Status 404 returned error can't find the container with id d7aa10f639853fee6ac2674ea06e2bd934619dbe9f92b6db5dd6552b10716065 Oct 30 20:16:58.314905 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:58.314864 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" event={"ID":"8ab4f322-4369-489b-abdf-82b2530a0b66","Type":"ContainerStarted","Data":"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85"} Oct 30 20:16:58.315128 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:58.314922 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" event={"ID":"8ab4f322-4369-489b-abdf-82b2530a0b66","Type":"ContainerStarted","Data":"d7aa10f639853fee6ac2674ea06e2bd934619dbe9f92b6db5dd6552b10716065"} Oct 30 20:16:58.349579 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:58.349537 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" podStartSLOduration=1.3495234090000001 podStartE2EDuration="1.349523409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:58.348713009 +0000 UTC m=+4326.060238374" watchObservedRunningTime="2026-10-30 20:16:58.349523409 +0000 UTC m=+4326.061048775" Oct 30 20:16:58.847714 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:16:58.847685 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:17:01.245078 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:01.245035 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:01.387116 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:01.386399 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:05.078408 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:05.078323 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:05.095190 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:17:05.095155 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:17:05.247437 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:05.247403 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:07.847954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:07.847921 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:17:07.860461 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:07.860434 4990 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:17:08.401247 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:08.401219 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:17:34.645678 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.645644 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7"] Oct 30 20:17:34.648961 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.648942 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.660986 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.660963 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:17:34.670928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.670897 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:34.671051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.670989 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:17:34.671051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.671031 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:17:34.673953 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.673932 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7"] Oct 30 20:17:34.749897 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.749863 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pzx\" (UniqueName: \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.750136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.749927 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.750136 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.750031 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.851528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.851495 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.851739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.851539 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.851739 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.851715 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pzx\" (UniqueName: \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.852200 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.852179 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.853813 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.853786 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.868980 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.868954 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pzx\" (UniqueName: \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\") pod \"maas-api-revoke-sub-452b6d7b-7tpj7\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:34.962409 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:34.962334 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:35.095711 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:17:35.095675 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:17:35.257442 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:35.257370 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7"] Oct 30 20:17:35.261049 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:17:35.261019 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaffa3d7_3a8c_4871_91ac_11afe6dbed14.slice/crio-48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a WatchSource:0}: Error finding container 48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a: Status 404 returned error can't find the container with id 48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a Oct 30 20:17:35.536746 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:35.536654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" event={"ID":"daffa3d7-3a8c-4871-91ac-11afe6dbed14","Type":"ContainerStarted","Data":"48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a"} Oct 30 20:17:36.547155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:36.547043 4990 generic.go:356] "Generic (PLEG): container finished" podID="daffa3d7-3a8c-4871-91ac-11afe6dbed14" containerID="48d48576f27aedf700f66448fb73dba2a129d165ac2d81cfdfd51ce96556f92e" exitCode=0 Oct 30 20:17:36.547155 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:36.547131 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" event={"ID":"daffa3d7-3a8c-4871-91ac-11afe6dbed14","Type":"ContainerDied","Data":"48d48576f27aedf700f66448fb73dba2a129d165ac2d81cfdfd51ce96556f92e"} Oct 30 20:17:37.974583 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:37.974267 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:38.050527 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.050487 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\") pod \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " Oct 30 20:17:38.051238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.051213 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\" (UniqueName: \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\") pod \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " Oct 30 20:17:38.051449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.051429 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\") pod \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\" (UID: \"daffa3d7-3a8c-4871-91ac-11afe6dbed14\") " Oct 30 20:17:38.052499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.051125 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca" pod "daffa3d7-3a8c-4871-91ac-11afe6dbed14" (UID: "daffa3d7-3a8c-4871-91ac-11afe6dbed14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:38.063723 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.063677 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx" pod "daffa3d7-3a8c-4871-91ac-11afe6dbed14" (UID: "daffa3d7-3a8c-4871-91ac-11afe6dbed14"). InnerVolumeSpecName "kube-api-access-z7pzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:38.096365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.096305 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca" pod "daffa3d7-3a8c-4871-91ac-11afe6dbed14" (UID: "daffa3d7-3a8c-4871-91ac-11afe6dbed14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:38.156045 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.155760 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:17:38.156045 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.155804 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pzx\" (UniqueName: \"kubernetes.io/projected/daffa3d7-3a8c-4871-91ac-11afe6dbed14-kube-api-access-z7pzx\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:17:38.156045 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.155822 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daffa3d7-3a8c-4871-91ac-11afe6dbed14-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:17:38.565203 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.565162 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" event={"ID":"daffa3d7-3a8c-4871-91ac-11afe6dbed14","Type":"ContainerDied","Data":"48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a"} Oct 30 20:17:38.565413 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.565230 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48182b9acb6d06fe8e45ddbfa94ee3148929af99f3fb956721521a9fccf78a9a" Oct 30 20:17:38.565413 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:38.565297 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7" Oct 30 20:17:55.811662 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:55.811621 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:17:55.902581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:55.902544 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:58.811105 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:58.811048 4990 scope.go:122] "RemoveContainer" containerID="7e905d378c40376bcb0d10f744ba27f884416ed7c13145c991341de742ad37e6" Oct 30 20:17:58.820451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:58.820427 4990 scope.go:122] "RemoveContainer" containerID="ea62a110d04180e1b2f4c9c64fb2a964e82de874bf7aef29a7fd9510d3dc68b1" Oct 30 20:17:58.830687 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:17:58.830657 4990 scope.go:122] "RemoveContainer" containerID="99bf873d39d2cef68444e1f997c4ab655ecc3b1237ac630a82a9c6fb80e74907" Oct 30 20:18:01.756464 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:01.756426 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:18:01.849538 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:01.849494 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:05.124332 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:18:05.124288 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:18:05.310721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.310682 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:05.645346 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.645306 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64"] Oct 30 20:18:05.645917 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.645894 4990 cpu_manager.go:414] "containerMap: removing container" podUID="daffa3d7-3a8c-4871-91ac-11afe6dbed14" containerName="revoke-keys" Oct 30 20:18:05.645974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.645922 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daffa3d7-3a8c-4871-91ac-11afe6dbed14" containerName="revoke-keys" Oct 30 20:18:05.646026 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.646013 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="daffa3d7-3a8c-4871-91ac-11afe6dbed14" containerName="revoke-keys" Oct 30 20:18:05.651260 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.651239 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.676176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.675890 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:05.676176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.675899 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:18:05.676176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.675956 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:18:05.676176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.676005 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:18:05.678605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.678579 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64"] Oct 30 20:18:05.804184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.804137 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsf7x\" (UniqueName: \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.804517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.804273 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.804517 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.804380 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.905830 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.905735 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.905830 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.905785 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsf7x\" (UniqueName: \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.905830 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.905824 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.906487 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.906465 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.908381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.908350 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.923412 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.923389 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsf7x\" (UniqueName: \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\") pod \"maas-api-revoke-sub-3e31a506-z6p64\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:05.964366 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:05.964339 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:06.264061 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:06.263966 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64"] Oct 30 20:18:06.268562 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:18:06.268531 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fcf272_1635_44e2_ae42_8ea08ac4e3ab.slice/crio-f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1 WatchSource:0}: Error finding container f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1: Status 404 returned error can't find the container with id f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1 Oct 30 20:18:06.764451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:06.764406 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" event={"ID":"26fcf272-1635-44e2-ae42-8ea08ac4e3ab","Type":"ContainerStarted","Data":"c1582c9221d2d2c430be5636313ffea9d4bcd3d3385416bbb6054830f006a218"} Oct 30 20:18:06.764663 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:06.764480 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" event={"ID":"26fcf272-1635-44e2-ae42-8ea08ac4e3ab","Type":"ContainerStarted","Data":"f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1"} Oct 30 20:18:06.797287 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:06.797221 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" podStartSLOduration=1.7972016480000002 podStartE2EDuration="1.797201648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:06.796950672 +0000 UTC m=+4394.508476038" watchObservedRunningTime="2026-10-30 20:18:06.797201648 +0000 UTC m=+4394.508727015" Oct 30 20:18:07.804381 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:07.804336 4990 generic.go:356] "Generic (PLEG): container finished" podID="26fcf272-1635-44e2-ae42-8ea08ac4e3ab" containerID="c1582c9221d2d2c430be5636313ffea9d4bcd3d3385416bbb6054830f006a218" exitCode=0 Oct 30 20:18:07.805083 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:07.805053 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" event={"ID":"26fcf272-1635-44e2-ae42-8ea08ac4e3ab","Type":"ContainerDied","Data":"c1582c9221d2d2c430be5636313ffea9d4bcd3d3385416bbb6054830f006a218"} Oct 30 20:18:09.134017 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.133989 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:09.262685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.262649 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\") pod \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " Oct 30 20:18:09.262893 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.262731 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\" (UniqueName: \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\") pod \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " Oct 30 20:18:09.262893 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.262792 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\") pod \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\" (UID: \"26fcf272-1635-44e2-ae42-8ea08ac4e3ab\") " Oct 30 20:18:09.263231 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.263196 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca" pod "26fcf272-1635-44e2-ae42-8ea08ac4e3ab" (UID: "26fcf272-1635-44e2-ae42-8ea08ac4e3ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:09.264864 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.264836 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca" pod "26fcf272-1635-44e2-ae42-8ea08ac4e3ab" (UID: "26fcf272-1635-44e2-ae42-8ea08ac4e3ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:09.264974 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.264890 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x" pod "26fcf272-1635-44e2-ae42-8ea08ac4e3ab" (UID: "26fcf272-1635-44e2-ae42-8ea08ac4e3ab"). InnerVolumeSpecName "kube-api-access-tsf7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:09.364273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.364197 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsf7x\" (UniqueName: \"kubernetes.io/projected/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-kube-api-access-tsf7x\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.364273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.364227 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.364273 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.364237 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fcf272-1635-44e2-ae42-8ea08ac4e3ab-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:09.843176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.843143 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" Oct 30 20:18:09.843176 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.843154 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64" event={"ID":"26fcf272-1635-44e2-ae42-8ea08ac4e3ab","Type":"ContainerDied","Data":"f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1"} Oct 30 20:18:09.843455 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:09.843203 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46d5d20cccb30012db3934e57773578196aa3058d1c31a506e3a1f44a7761a1" Oct 30 20:18:10.831867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:10.831824 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" podUID="c57445e7-af71-4aff-a867-82c510c9261e" containerName="main" containerID="cri-o://b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0" gracePeriod=45 Oct 30 20:18:10.838828 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:10.838702 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:10.988253 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:10.988209 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:11.096145 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.096080 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:18:11.182494 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182464 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.182689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182512 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\" (UniqueName: \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.182689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182546 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.182689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182576 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.182689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182640 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.182689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182665 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\") pod \"c57445e7-af71-4aff-a867-82c510c9261e\" (UID: \"c57445e7-af71-4aff-a867-82c510c9261e\") " Oct 30 20:18:11.183007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182857 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:11.183007 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.182978 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.183171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.183019 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:11.183171 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.183056 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:11.184982 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.184957 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:11.184982 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.184975 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:11.185158 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.185022 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp" pod "c57445e7-af71-4aff-a867-82c510c9261e" (UID: "c57445e7-af71-4aff-a867-82c510c9261e"). InnerVolumeSpecName "kube-api-access-rzfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:11.284426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.284391 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.284426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.284421 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.284426 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.284432 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57445e7-af71-4aff-a867-82c510c9261e-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.284671 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.284440 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57445e7-af71-4aff-a867-82c510c9261e-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.284671 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.284449 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfsp\" (UniqueName: \"kubernetes.io/projected/c57445e7-af71-4aff-a867-82c510c9261e-kube-api-access-rzfsp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:11.855306 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.855272 4990 generic.go:356] "Generic (PLEG): container finished" podID="c57445e7-af71-4aff-a867-82c510c9261e" containerID="b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0" exitCode=0 Oct 30 20:18:11.855878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.855319 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" Oct 30 20:18:11.855878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.855357 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" event={"ID":"c57445e7-af71-4aff-a867-82c510c9261e","Type":"ContainerDied","Data":"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0"} Oct 30 20:18:11.855878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.855412 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2" event={"ID":"c57445e7-af71-4aff-a867-82c510c9261e","Type":"ContainerDied","Data":"d1a6a26e77fed93d4aa42416014f61e1738882f4da17120f58ced5e0888b89eb"} Oct 30 20:18:11.855878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.855438 4990 scope.go:122] "RemoveContainer" containerID="b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0" Oct 30 20:18:11.866367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.866347 4990 scope.go:122] "RemoveContainer" containerID="b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0" Oct 30 20:18:11.866657 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:18:11.866638 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0\": container with ID starting with b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0 not found: ID does not exist" containerID="b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0" Oct 30 20:18:11.866709 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.866666 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0"} err="failed to get container status \"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0\": rpc error: code = NotFound desc = could not find container \"b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0\": container with ID starting with b21cd9932932885964b89426df030976537d70746747574d1ce658248d2fa9c0 not found: ID does not exist" Oct 30 20:18:11.899728 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.899701 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:18:11.909067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:11.909043 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-41fd0cc3-kserve-755bfc8f85-wtpl2"] Oct 30 20:18:12.400382 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:12.400348 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57445e7-af71-4aff-a867-82c510c9261e" path="/var/lib/kubelet/pods/c57445e7-af71-4aff-a867-82c510c9261e/volumes" Oct 30 20:18:16.776988 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:16.776942 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" podUID="8ab4f322-4369-489b-abdf-82b2530a0b66" containerName="main" containerID="cri-o://37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85" gracePeriod=45 Oct 30 20:18:17.052170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.052138 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:18:17.146218 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146188 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146277 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146313 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146343 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\" (UniqueName: \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146410 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146373 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146428 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\") pod \"8ab4f322-4369-489b-abdf-82b2530a0b66\" (UID: \"8ab4f322-4369-489b-abdf-82b2530a0b66\") " Oct 30 20:18:17.146716 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146629 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.146774 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146763 4990 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-model-cache\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.146872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146843 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.147006 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.146975 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.148559 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.148529 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:17.148781 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.148753 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.149219 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.149150 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk" pod "8ab4f322-4369-489b-abdf-82b2530a0b66" (UID: "8ab4f322-4369-489b-abdf-82b2530a0b66"). InnerVolumeSpecName "kube-api-access-rz9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:17.247549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.247493 4990 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-tmp-dir\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.247549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.247536 4990 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-home\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.247549 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.247551 4990 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab4f322-4369-489b-abdf-82b2530a0b66-dshm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.247954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.247564 4990 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab4f322-4369-489b-abdf-82b2530a0b66-tls-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.247954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.247580 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9pk\" (UniqueName: \"kubernetes.io/projected/8ab4f322-4369-489b-abdf-82b2530a0b66-kube-api-access-rz9pk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:17.890605 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.890570 4990 generic.go:356] "Generic (PLEG): container finished" podID="8ab4f322-4369-489b-abdf-82b2530a0b66" containerID="37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85" exitCode=0 Oct 30 20:18:17.891053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.890635 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" Oct 30 20:18:17.891053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.890656 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" event={"ID":"8ab4f322-4369-489b-abdf-82b2530a0b66","Type":"ContainerDied","Data":"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85"} Oct 30 20:18:17.891053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.890700 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk" event={"ID":"8ab4f322-4369-489b-abdf-82b2530a0b66","Type":"ContainerDied","Data":"d7aa10f639853fee6ac2674ea06e2bd934619dbe9f92b6db5dd6552b10716065"} Oct 30 20:18:17.891053 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.890717 4990 scope.go:122] "RemoveContainer" containerID="37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85" Oct 30 20:18:17.900339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.900317 4990 scope.go:122] "RemoveContainer" containerID="37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85" Oct 30 20:18:17.900583 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:18:17.900569 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85\": container with ID starting with 37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85 not found: ID does not exist" containerID="37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85" Oct 30 20:18:17.900635 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.900592 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85"} err="failed to get container status \"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85\": rpc error: code = NotFound desc = could not find container \"37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85\": container with ID starting with 37f8b8b96749af3e11fe4f749f50ad73218a48c94ad3368e000171eb90a78d85 not found: ID does not exist" Oct 30 20:18:17.933559 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.933530 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:18:17.942970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:17.942938 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-41fd0cc3-kserve-79b4485b9-jhhrk"] Oct 30 20:18:18.399334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:18.399299 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab4f322-4369-489b-abdf-82b2530a0b66" path="/var/lib/kubelet/pods/8ab4f322-4369-489b-abdf-82b2530a0b66/volumes" Oct 30 20:18:25.967721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.967678 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns"] Oct 30 20:18:25.968278 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968257 4990 cpu_manager.go:414] "containerMap: removing container" podUID="26fcf272-1635-44e2-ae42-8ea08ac4e3ab" containerName="revoke-keys" Oct 30 20:18:25.968323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968281 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fcf272-1635-44e2-ae42-8ea08ac4e3ab" containerName="revoke-keys" Oct 30 20:18:25.968323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968301 4990 cpu_manager.go:414] "containerMap: removing container" podUID="c57445e7-af71-4aff-a867-82c510c9261e" containerName="main" Oct 30 20:18:25.968323 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968309 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57445e7-af71-4aff-a867-82c510c9261e" containerName="main" Oct 30 20:18:25.968429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968336 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8ab4f322-4369-489b-abdf-82b2530a0b66" containerName="main" Oct 30 20:18:25.968429 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968345 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab4f322-4369-489b-abdf-82b2530a0b66" containerName="main" Oct 30 20:18:25.968492 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968452 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fcf272-1635-44e2-ae42-8ea08ac4e3ab" containerName="revoke-keys" Oct 30 20:18:25.968492 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968466 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57445e7-af71-4aff-a867-82c510c9261e" containerName="main" Oct 30 20:18:25.968492 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.968478 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab4f322-4369-489b-abdf-82b2530a0b66" containerName="main" Oct 30 20:18:25.973000 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.972979 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:25.989701 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:25.989667 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns"] Oct 30 20:18:26.004358 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.004331 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:18:26.004498 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.004359 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:18:26.014310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.014284 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:26.014310 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.014294 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:18:26.060529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.060495 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.060692 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.060541 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.060692 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.060565 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkfm\" (UniqueName: \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.161581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.161546 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.161753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.161590 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.161753 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.161613 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkfm\" (UniqueName: \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.162353 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.162327 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.163888 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.163867 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.178924 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.178898 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkfm\" (UniqueName: \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\") pod \"maas-api-revoke-sub-ff817162-mk7ns\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.285812 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.285728 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:26.583600 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.583522 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns"] Oct 30 20:18:26.588481 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:18:26.588455 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1b3917_b7a4_462b_823a_5848bfe9cbbd.slice/crio-f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b WatchSource:0}: Error finding container f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b: Status 404 returned error can't find the container with id f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b Oct 30 20:18:26.954374 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.954337 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" event={"ID":"7a1b3917-b7a4-462b-823a-5848bfe9cbbd","Type":"ContainerStarted","Data":"b0473a6e40123fb9bd2e1ac3a1a8fd555ca55dd633174ddb78c5a916f5fa2605"} Oct 30 20:18:26.954541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.954401 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" event={"ID":"7a1b3917-b7a4-462b-823a-5848bfe9cbbd","Type":"ContainerStarted","Data":"f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b"} Oct 30 20:18:26.988005 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:26.987950 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" podStartSLOduration=1.987937578 podStartE2EDuration="1.987937578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:26.987360227 +0000 UTC m=+4414.698885593" watchObservedRunningTime="2026-10-30 20:18:26.987937578 +0000 UTC m=+4414.699462944" Oct 30 20:18:27.961138 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:27.961085 4990 generic.go:356] "Generic (PLEG): container finished" podID="7a1b3917-b7a4-462b-823a-5848bfe9cbbd" containerID="b0473a6e40123fb9bd2e1ac3a1a8fd555ca55dd633174ddb78c5a916f5fa2605" exitCode=0 Oct 30 20:18:27.961138 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:27.961121 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" event={"ID":"7a1b3917-b7a4-462b-823a-5848bfe9cbbd","Type":"ContainerDied","Data":"b0473a6e40123fb9bd2e1ac3a1a8fd555ca55dd633174ddb78c5a916f5fa2605"} Oct 30 20:18:29.096665 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.096640 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:29.194154 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.194120 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\") pod \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " Oct 30 20:18:29.194356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.194189 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\" (UniqueName: \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\") pod \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " Oct 30 20:18:29.194356 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.194327 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\") pod \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\" (UID: \"7a1b3917-b7a4-462b-823a-5848bfe9cbbd\") " Oct 30 20:18:29.194704 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.194673 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca" pod "7a1b3917-b7a4-462b-823a-5848bfe9cbbd" (UID: "7a1b3917-b7a4-462b-823a-5848bfe9cbbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:29.196157 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.196127 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca" pod "7a1b3917-b7a4-462b-823a-5848bfe9cbbd" (UID: "7a1b3917-b7a4-462b-823a-5848bfe9cbbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:29.196334 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.196314 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm" pod "7a1b3917-b7a4-462b-823a-5848bfe9cbbd" (UID: "7a1b3917-b7a4-462b-823a-5848bfe9cbbd"). InnerVolumeSpecName "kube-api-access-ghkfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:29.295450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.295373 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-openshift-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:29.295450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.295405 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-maas-api-xks-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:29.295450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.295416 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghkfm\" (UniqueName: \"kubernetes.io/projected/7a1b3917-b7a4-462b-823a-5848bfe9cbbd-kube-api-access-ghkfm\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:18:29.973751 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.973716 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" Oct 30 20:18:29.973751 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.973742 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns" event={"ID":"7a1b3917-b7a4-462b-823a-5848bfe9cbbd","Type":"ContainerDied","Data":"f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b"} Oct 30 20:18:29.973990 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:29.973795 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b4462ee16031c2540f03716bdfa3e37944b757547c7c39fe124802f1545a0b" Oct 30 20:18:35.026497 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:35.026421 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq"] Oct 30 20:18:35.035944 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:35.035918 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drvcq"] Oct 30 20:18:35.101422 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:18:35.101386 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:18:36.399599 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:36.399567 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127fb895-909b-4724-a6b4-05aeecd3a148" path="/var/lib/kubelet/pods/127fb895-909b-4724-a6b4-05aeecd3a148/volumes" Oct 30 20:18:51.028162 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:51.028124 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52"] Oct 30 20:18:51.037337 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:51.037305 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nsb52"] Oct 30 20:18:52.399848 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:18:52.399815 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df1d20c-3991-431b-b493-6a74ea337089" path="/var/lib/kubelet/pods/3df1d20c-3991-431b-b493-6a74ea337089/volumes" Oct 30 20:19:05.095408 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:19:05.095363 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:19:05.456451 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:05.456337 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:08.560200 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.560164 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:08.560603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.560533 4990 cpu_manager.go:414] "containerMap: removing container" podUID="7a1b3917-b7a4-462b-823a-5848bfe9cbbd" containerName="revoke-keys" Oct 30 20:19:08.560603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.560544 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1b3917-b7a4-462b-823a-5848bfe9cbbd" containerName="revoke-keys" Oct 30 20:19:08.560603 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.560602 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1b3917-b7a4-462b-823a-5848bfe9cbbd" containerName="revoke-keys" Oct 30 20:19:08.563659 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.563629 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.587292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.587239 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:19:08.587292 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.587239 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:19:08.587531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.587506 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:19:08.587531 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.587507 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:19:08.587703 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.587506 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r84xv\"" Oct 30 20:19:08.590817 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.590793 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:08.653652 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.653624 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.653821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.653665 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.653821 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.653697 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpltp\" (UniqueName: \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.653907 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.653843 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.754654 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.754617 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.754862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.754676 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.754862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.754696 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.754862 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.754724 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpltp\" (UniqueName: \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.755459 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.755430 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.757499 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.757468 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.757622 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.757587 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.772485 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.772457 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpltp\" (UniqueName: \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\") pod \"maas-api-5c48877c75-t2lnr\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:08.876009 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:08.875930 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:09.174720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:09.174644 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:09.178357 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:19:09.178327 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70863ead_84b0_45ad_b88f_9d5019a90da1.slice/crio-3efdbd3b6de781c1c388c8eab060f128fc719d1b862897ea1a30b68dd7202573 WatchSource:0}: Error finding container 3efdbd3b6de781c1c388c8eab060f128fc719d1b862897ea1a30b68dd7202573: Status 404 returned error can't find the container with id 3efdbd3b6de781c1c388c8eab060f128fc719d1b862897ea1a30b68dd7202573 Oct 30 20:19:09.179744 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:09.179717 4990 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:19:10.187129 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:10.187074 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" event={"ID":"70863ead-84b0-45ad-b88f-9d5019a90da1","Type":"ContainerStarted","Data":"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85"} Oct 30 20:19:10.187570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:10.187157 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" event={"ID":"70863ead-84b0-45ad-b88f-9d5019a90da1","Type":"ContainerStarted","Data":"3efdbd3b6de781c1c388c8eab060f128fc719d1b862897ea1a30b68dd7202573"} Oct 30 20:19:10.187570 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:10.187253 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:10.222645 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:10.222591 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" podStartSLOduration=1.7803287110000001 podStartE2EDuration="2.222576595s" totalImagesPullingTime="442.247884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:10.22071256 +0000 UTC m=+4457.932237937" watchObservedRunningTime="2026-10-30 20:19:10.222576595 +0000 UTC m=+4457.934101962" Oct 30 20:19:12.895910 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:12.895866 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:16.199246 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.199207 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:16.200514 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.200487 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:16.218362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.218315 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" podUID="70863ead-84b0-45ad-b88f-9d5019a90da1" containerName="maas-api" containerID="cri-o://a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85" gracePeriod=30 Oct 30 20:19:16.234067 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.234043 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-l2kt5"] Oct 30 20:19:16.238213 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.238196 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.255401 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.255381 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-l2kt5"] Oct 30 20:19:16.426800 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.426759 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-metrics-certs\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.427000 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.426822 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtml\" (UniqueName: \"kubernetes.io/projected/2e95625f-6920-4d47-bc36-e78ae777b7fb-kube-api-access-hqtml\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.427000 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.426951 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-maas-api-tls\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.427140 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.426997 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e95625f-6920-4d47-bc36-e78ae777b7fb-openshift-service-ca\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.466165 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.466138 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:16.528454 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.528421 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-maas-api-tls\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.528454 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.528454 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e95625f-6920-4d47-bc36-e78ae777b7fb-openshift-service-ca\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.528685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.528496 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-metrics-certs\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.528685 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.528517 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtml\" (UniqueName: \"kubernetes.io/projected/2e95625f-6920-4d47-bc36-e78ae777b7fb-kube-api-access-hqtml\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.529294 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.529267 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e95625f-6920-4d47-bc36-e78ae777b7fb-openshift-service-ca\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.530977 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.530947 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-maas-api-tls\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.531113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.530991 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e95625f-6920-4d47-bc36-e78ae777b7fb-metrics-certs\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.544926 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.544904 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtml\" (UniqueName: \"kubernetes.io/projected/2e95625f-6920-4d47-bc36-e78ae777b7fb-kube-api-access-hqtml\") pod \"maas-api-595cc648c-l2kt5\" (UID: \"2e95625f-6920-4d47-bc36-e78ae777b7fb\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.591781 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.591751 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:16.629001 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.628972 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\") pod \"70863ead-84b0-45ad-b88f-9d5019a90da1\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " Oct 30 20:19:16.629167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.629086 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\" (UniqueName: \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\") pod \"70863ead-84b0-45ad-b88f-9d5019a90da1\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " Oct 30 20:19:16.629167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.629141 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\") pod \"70863ead-84b0-45ad-b88f-9d5019a90da1\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " Oct 30 20:19:16.629367 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.629194 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\") pod \"70863ead-84b0-45ad-b88f-9d5019a90da1\" (UID: \"70863ead-84b0-45ad-b88f-9d5019a90da1\") " Oct 30 20:19:16.629591 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.629565 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca" pod "70863ead-84b0-45ad-b88f-9d5019a90da1" (UID: "70863ead-84b0-45ad-b88f-9d5019a90da1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:16.631683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.631639 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp" pod "70863ead-84b0-45ad-b88f-9d5019a90da1" (UID: "70863ead-84b0-45ad-b88f-9d5019a90da1"). InnerVolumeSpecName "kube-api-access-wpltp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:16.631954 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.631927 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls" pod "70863ead-84b0-45ad-b88f-9d5019a90da1" (UID: "70863ead-84b0-45ad-b88f-9d5019a90da1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:16.632530 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.632506 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs" pod "70863ead-84b0-45ad-b88f-9d5019a90da1" (UID: "70863ead-84b0-45ad-b88f-9d5019a90da1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:16.730568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.730485 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpltp\" (UniqueName: \"kubernetes.io/projected/70863ead-84b0-45ad-b88f-9d5019a90da1-kube-api-access-wpltp\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:19:16.730568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.730518 4990 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70863ead-84b0-45ad-b88f-9d5019a90da1-openshift-service-ca\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:19:16.730568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.730530 4990 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-metrics-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:19:16.730568 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.730545 4990 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70863ead-84b0-45ad-b88f-9d5019a90da1-maas-api-tls\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:19:16.891286 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:16.891257 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-l2kt5"] Oct 30 20:19:16.893457 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:19:16.893428 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e95625f_6920_4d47_bc36_e78ae777b7fb.slice/crio-1eb1c28fcc983234ee0c6b6762ed79a399d02ab45e3ac1df38eb491a43fbb7cf WatchSource:0}: Error finding container 1eb1c28fcc983234ee0c6b6762ed79a399d02ab45e3ac1df38eb491a43fbb7cf: Status 404 returned error can't find the container with id 1eb1c28fcc983234ee0c6b6762ed79a399d02ab45e3ac1df38eb491a43fbb7cf Oct 30 20:19:17.224389 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.224348 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" event={"ID":"2e95625f-6920-4d47-bc36-e78ae777b7fb","Type":"ContainerStarted","Data":"1eb1c28fcc983234ee0c6b6762ed79a399d02ab45e3ac1df38eb491a43fbb7cf"} Oct 30 20:19:17.225820 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.225790 4990 generic.go:356] "Generic (PLEG): container finished" podID="70863ead-84b0-45ad-b88f-9d5019a90da1" containerID="a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85" exitCode=0 Oct 30 20:19:17.225960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.225841 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" event={"ID":"70863ead-84b0-45ad-b88f-9d5019a90da1","Type":"ContainerDied","Data":"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85"} Oct 30 20:19:17.225960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.225849 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" Oct 30 20:19:17.225960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.225887 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr" event={"ID":"70863ead-84b0-45ad-b88f-9d5019a90da1","Type":"ContainerDied","Data":"3efdbd3b6de781c1c388c8eab060f128fc719d1b862897ea1a30b68dd7202573"} Oct 30 20:19:17.225960 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.225916 4990 scope.go:122] "RemoveContainer" containerID="a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85" Oct 30 20:19:17.244030 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.244011 4990 scope.go:122] "RemoveContainer" containerID="a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85" Oct 30 20:19:17.244305 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:19:17.244287 4990 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85\": container with ID starting with a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85 not found: ID does not exist" containerID="a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85" Oct 30 20:19:17.244361 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.244314 4990 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85"} err="failed to get container status \"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85\": rpc error: code = NotFound desc = could not find container \"a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85\": container with ID starting with a9457a76ec72f5b9dc7b58c7c8a2bc54e4fcd0464c174acfb79fbdef78c72b85 not found: ID does not exist" Oct 30 20:19:17.275239 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.275197 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:17.277796 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:17.277769 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-t2lnr"] Oct 30 20:19:18.248118 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:18.247516 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" event={"ID":"2e95625f-6920-4d47-bc36-e78ae777b7fb","Type":"ContainerStarted","Data":"4c89c310b4edae8212a47dad7bb4f9c355091b0e82ec398d81936018b18dc61d"} Oct 30 20:19:18.248651 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:18.248474 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:18.289894 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:18.289824 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" podStartSLOduration=1.951633829 podStartE2EDuration="2.289804509s" totalImagesPullingTime="338.17068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:18.288461144 +0000 UTC m=+4465.999986520" watchObservedRunningTime="2026-10-30 20:19:18.289804509 +0000 UTC m=+4466.001329877" Oct 30 20:19:18.422395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:18.422355 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70863ead-84b0-45ad-b88f-9d5019a90da1" path="/var/lib/kubelet/pods/70863ead-84b0-45ad-b88f-9d5019a90da1/volumes" Oct 30 20:19:25.283032 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:25.282296 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-l2kt5" Oct 30 20:19:35.095601 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:19:35.095563 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:19:54.840158 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:54.840128 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:19:54.849858 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:54.849836 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:19:54.864173 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:54.864149 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:19:54.873761 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:54.873741 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:19:56.905519 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:56.905461 4990 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-68.ec2.internal" podUID="c1a30bdfa789f3bccffc4563d5bc301a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 20:19:58.886473 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:58.886440 4990 scope.go:122] "RemoveContainer" containerID="2ef931655e6e44c464f187a7ad0dceb0088b2aff7b578e4558fb58b065c5ac76" Oct 30 20:19:58.895298 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:19:58.895278 4990 scope.go:122] "RemoveContainer" containerID="be973e35d0f52f5d47196efeb4565737e7c66f6dc6982185b97c73ab17fbbc8f" Oct 30 20:20:05.096702 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:20:05.096616 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:20:05.193492 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:05.193455 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56dcb668cc-fh7dx_cda75769-7027-40c0-8288-b1dfd4ddfe8a/authorino/0.log" Oct 30 20:20:05.729997 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:05.729956 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.027970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:32.027934 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4"] Oct 30 20:20:32.037319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:32.037286 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a426a1b-rhrc4"] Oct 30 20:20:32.399813 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:32.399776 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5" path="/var/lib/kubelet/pods/4b6a7fcf-c27f-468f-ac9f-b09dff40f1b5/volumes" Oct 30 20:20:35.095687 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:20:35.095648 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:20:38.343186 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:38.343155 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/manager/0.log" Oct 30 20:20:38.415528 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:38.415501 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/copy-manifests/0.log" Oct 30 20:20:39.625205 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:39.625170 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-l2kt5_2e95625f-6920-4d47-bc36-e78ae777b7fb/maas-api/0.log" Oct 30 20:20:40.220878 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:40.220849 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-z6p64_26fcf272-1635-44e2-ae42-8ea08ac4e3ab/revoke-keys/0.log" Oct 30 20:20:40.416490 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:40.416459 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-7tpj7_daffa3d7-3a8c-4871-91ac-11afe6dbed14/revoke-keys/0.log" Oct 30 20:20:40.987565 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:40.987535 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff817162-mk7ns_7a1b3917-b7a4-462b-823a-5848bfe9cbbd/revoke-keys/0.log" Oct 30 20:20:42.727676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:42.727642 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56dcb668cc-fh7dx_cda75769-7027-40c0-8288-b1dfd4ddfe8a/authorino/0.log" Oct 30 20:20:43.202027 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:43.201993 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xrlsb_e7a72d51-4726-4114-96de-9ec15e1ab570/manager/0.log" Oct 30 20:20:43.384643 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:43.384614 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jqxt6_e3f626ef-bed4-4974-9ba4-7778435cee18/kuadrant-console-plugin/0.log" Oct 30 20:20:43.601250 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:43.601215 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qpmtp_ad2b1fe0-8a6f-402b-a834-a6410a77fc72/manager/0.log" Oct 30 20:20:45.222810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:45.222777 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-dbd6g_83a243fd-8ba3-4e72-9760-6583b0458d47/kube-auth-proxy/0.log" Oct 30 20:20:45.591002 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:45.590967 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8l6gm_9d25c5e7-7376-49ac-bf09-25e92359d510/istio-proxy/0.log" Oct 30 20:20:46.009564 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:46.009479 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-czfvt_fcb08303-e74d-4246-8124-5c716d720da5/istio-proxy/0.log" Oct 30 20:20:47.249104 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:20:47.249071 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-vv8z5_8413d9c9-a94c-4b84-ad92-eee68b5beac4/istio-proxy/0.log" Oct 30 20:21:05.095285 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:21:05.095249 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:21:05.852403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:05.852366 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:08.985783 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:08.985754 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm_98ae7206-c9c4-4e25-ae72-dcb9ff5436b5/main/0.log" Oct 30 20:21:09.410711 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:09.410676 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p8tsx_b37ddf4c-70f5-421f-864e-7305d672b1a4/main/0.log" Oct 30 20:21:09.612991 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:09.612963 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ndzw7_97f8168e-4fa4-4e31-be2a-caecba1f1a46/main/0.log" Oct 30 20:21:10.232948 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:10.232918 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp_052bfe48-6728-4136-93e2-7090d3d90481/main/0.log" Oct 30 20:21:21.226351 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:21.226307 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56dcb668cc-fh7dx_cda75769-7027-40c0-8288-b1dfd4ddfe8a/authorino/0.log" Oct 30 20:21:23.297582 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:23.297546 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:23.433460 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:23.433425 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.094914 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:21:35.094831 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:21:42.833041 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:42.833010 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/copy-manifests/0.log" Oct 30 20:21:42.873806 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:42.873774 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/manager/0.log" Oct 30 20:21:43.679581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:43.679548 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-l2kt5_2e95625f-6920-4d47-bc36-e78ae777b7fb/maas-api/0.log" Oct 30 20:21:54.323781 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:54.323743 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-z6p64_26fcf272-1635-44e2-ae42-8ea08ac4e3ab/revoke-keys/0.log" Oct 30 20:21:54.522040 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:54.522008 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-7tpj7_daffa3d7-3a8c-4871-91ac-11afe6dbed14/revoke-keys/0.log" Oct 30 20:21:55.096874 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:55.096834 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff817162-mk7ns_7a1b3917-b7a4-462b-823a-5848bfe9cbbd/revoke-keys/0.log" Oct 30 20:21:56.473248 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:56.473207 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56dcb668cc-fh7dx_cda75769-7027-40c0-8288-b1dfd4ddfe8a/authorino/0.log" Oct 30 20:21:56.949729 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:56.949675 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xrlsb_e7a72d51-4726-4114-96de-9ec15e1ab570/manager/0.log" Oct 30 20:21:57.148285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:57.148256 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jqxt6_e3f626ef-bed4-4974-9ba4-7778435cee18/kuadrant-console-plugin/0.log" Oct 30 20:21:57.352441 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:57.352409 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qpmtp_ad2b1fe0-8a6f-402b-a834-a6410a77fc72/manager/0.log" Oct 30 20:21:58.821154 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:58.821123 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-dbd6g_83a243fd-8ba3-4e72-9760-6583b0458d47/kube-auth-proxy/0.log" Oct 30 20:21:58.930166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:58.930131 4990 scope.go:122] "RemoveContainer" containerID="6de1d3251d6a44ae3dcc4e6f1c4b85fc9acf3f6a3b6c43dac6598174b4447cf7" Oct 30 20:21:58.941771 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:58.941745 4990 scope.go:122] "RemoveContainer" containerID="99f32c97ffe5dbd891720204258c6b54b5c6f708f5f976268c7b9fc3afda9220" Oct 30 20:21:59.203476 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:59.203388 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-8l6gm_9d25c5e7-7376-49ac-bf09-25e92359d510/istio-proxy/0.log" Oct 30 20:21:59.601586 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:21:59.601554 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-czfvt_fcb08303-e74d-4246-8124-5c716d720da5/istio-proxy/0.log" Oct 30 20:22:00.834779 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:00.834746 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-vv8z5_8413d9c9-a94c-4b84-ad92-eee68b5beac4/istio-proxy/0.log" Oct 30 20:22:02.212824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:02.212790 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-p2hlm_98ae7206-c9c4-4e25-ae72-dcb9ff5436b5/main/0.log" Oct 30 20:22:02.621039 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:02.621007 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p8tsx_b37ddf4c-70f5-421f-864e-7305d672b1a4/main/0.log" Oct 30 20:22:02.822131 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:02.822086 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-ndzw7_97f8168e-4fa4-4e31-be2a-caecba1f1a46/main/0.log" Oct 30 20:22:05.096014 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:22:05.095981 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:22:05.973132 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:05.973075 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.457634 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:13.457597 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fvtgp_052bfe48-6728-4136-93e2-7090d3d90481/main/0.log" Oct 30 20:22:34.737721 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:34.737683 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:34.878377 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:34.878342 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.095436 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:22:35.095402 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:22:39.029137 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:39.029102 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7"] Oct 30 20:22:39.037880 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:39.037854 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-7tpj7"] Oct 30 20:22:40.398676 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:22:40.398641 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daffa3d7-3a8c-4871-91ac-11afe6dbed14" path="/var/lib/kubelet/pods/daffa3d7-3a8c-4871-91ac-11afe6dbed14/volumes" Oct 30 20:23:05.096126 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:23:05.096026 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:23:06.084362 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:06.084328 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:06.220597 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:06.220556 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:09.024512 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:09.024456 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64"] Oct 30 20:23:09.033750 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:09.033722 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z6p64"] Oct 30 20:23:10.398742 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:10.398706 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fcf272-1635-44e2-ae42-8ea08ac4e3ab" path="/var/lib/kubelet/pods/26fcf272-1635-44e2-ae42-8ea08ac4e3ab/volumes" Oct 30 20:23:30.024738 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:30.024706 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns"] Oct 30 20:23:30.034551 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:30.034524 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff817162-mk7ns"] Oct 30 20:23:30.398939 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:30.398905 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1b3917-b7a4-462b-823a-5848bfe9cbbd" path="/var/lib/kubelet/pods/7a1b3917-b7a4-462b-823a-5848bfe9cbbd/volumes" Oct 30 20:23:33.376887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:33.376843 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:35.094739 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:23:35.094700 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:23:38.685856 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:38.685819 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:39.427061 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:39.427020 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:55.539733 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:55.539695 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:58.973811 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:23:58.973779 4990 scope.go:122] "RemoveContainer" containerID="48d48576f27aedf700f66448fb73dba2a129d165ac2d81cfdfd51ce96556f92e" Oct 30 20:24:05.094929 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:24:05.094895 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:24:06.208648 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:06.208604 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:35.095373 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:24:35.095293 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:24:53.211945 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:53.211901 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.880871 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:54.880752 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:24:54.890129 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:54.890078 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:24:54.899754 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:54.899735 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9k94l_01f6e335-5ec7-4638-a9d3-9ad719491a9e/extract/0.log" Oct 30 20:24:54.908044 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:54.908027 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:24:58.995324 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:58.995289 4990 scope.go:122] "RemoveContainer" containerID="c1582c9221d2d2c430be5636313ffea9d4bcd3d3385416bbb6054830f006a218" Oct 30 20:24:59.003872 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:24:59.003851 4990 scope.go:122] "RemoveContainer" containerID="b0473a6e40123fb9bd2e1ac3a1a8fd555ca55dd633174ddb78c5a916f5fa2605" Oct 30 20:25:05.094889 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:25:05.094847 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:25:05.857036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:05.856988 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:06.000139 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:06.000079 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:06.330034 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:06.330003 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:09.997035 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:09.997002 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9kwwp_fc2715ff-fc4b-46c6-89fe-f95275cfe46d/global-pull-secret-syncer/0.log" Oct 30 20:25:10.944404 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:10.944373 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" Oct 30 20:25:11.419976 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:11.419940 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-68.ec2.internal_c1a30bdfa789f3bccffc4563d5bc301a/haproxy/0.log" Oct 30 20:25:28.871167 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.871134 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:25:28.871669 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.871322 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" containerID="cri-o://8873c9a7f3b46720983d5143633ddef5be5b51840989130e9b12988d4af7202a" gracePeriod=30 Oct 30 20:25:28.883238 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.883198 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:25:28.883480 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.883436 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" containerID="cri-o://c595a274b8b270407bf2a10b7cbdb0f9b568f43d59ce9277a708854b5388bce5" gracePeriod=30 Oct 30 20:25:28.885173 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.885147 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:25:28.885336 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:28.885316 4990 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" containerID="cri-o://7550ad647610d548a80f62fea9176bceb0271f8ab3167870642166d983485119" gracePeriod=30 Oct 30 20:25:34.181529 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.181496 4990 generic.go:356] "Generic (PLEG): container finished" podID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerID="8873c9a7f3b46720983d5143633ddef5be5b51840989130e9b12988d4af7202a" exitCode=0 Oct 30 20:25:34.181855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.181571 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" event={"ID":"9d25c5e7-7376-49ac-bf09-25e92359d510","Type":"ContainerDied","Data":"8873c9a7f3b46720983d5143633ddef5be5b51840989130e9b12988d4af7202a"} Oct 30 20:25:34.181855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.181654 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" event={"ID":"9d25c5e7-7376-49ac-bf09-25e92359d510","Type":"ContainerDied","Data":"6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b"} Oct 30 20:25:34.181855 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.181683 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6583eaf90014691c8bdeb02487a01d90e05c5eea52e2fc7a58869bd096067a6b" Oct 30 20:25:34.182911 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.182890 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:25:34.183025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.182914 4990 generic.go:356] "Generic (PLEG): container finished" podID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerID="c595a274b8b270407bf2a10b7cbdb0f9b568f43d59ce9277a708854b5388bce5" exitCode=0 Oct 30 20:25:34.183025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.182945 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" event={"ID":"8413d9c9-a94c-4b84-ad92-eee68b5beac4","Type":"ContainerDied","Data":"c595a274b8b270407bf2a10b7cbdb0f9b568f43d59ce9277a708854b5388bce5"} Oct 30 20:25:34.183025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.182992 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" event={"ID":"8413d9c9-a94c-4b84-ad92-eee68b5beac4","Type":"ContainerDied","Data":"6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987"} Oct 30 20:25:34.183025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.183016 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6039206960e731772b043f6f340ae236be93a5d74e808b3bd84b40f8cf1a6987" Oct 30 20:25:34.184515 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.184495 4990 generic.go:356] "Generic (PLEG): container finished" podID="fcb08303-e74d-4246-8124-5c716d720da5" containerID="7550ad647610d548a80f62fea9176bceb0271f8ab3167870642166d983485119" exitCode=0 Oct 30 20:25:34.184626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.184564 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" event={"ID":"fcb08303-e74d-4246-8124-5c716d720da5","Type":"ContainerDied","Data":"7550ad647610d548a80f62fea9176bceb0271f8ab3167870642166d983485119"} Oct 30 20:25:34.184626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.184604 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" event={"ID":"fcb08303-e74d-4246-8124-5c716d720da5","Type":"ContainerDied","Data":"fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6"} Oct 30 20:25:34.184626 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.184620 4990 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0fb64b79f036cf832c6d9b10e7b4cd6d67fabd9cd3d36677cb45e9947555e6" Oct 30 20:25:34.186245 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.186223 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:25:34.196642 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.196621 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:25:34.214060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214034 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.214209 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214132 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.214276 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214207 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.214339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214279 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.214339 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214309 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214338 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214351 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214391 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214415 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214422 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.214465 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214452 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214483 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214523 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214561 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214557 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214599 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214646 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214679 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214712 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214753 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.214795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214795 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214828 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214834 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214884 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214933 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.214984 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215027 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215049 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215075 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215121 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215130 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215185 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\") pod \"9d25c5e7-7376-49ac-bf09-25e92359d510\" (UID: \"9d25c5e7-7376-49ac-bf09-25e92359d510\") " Oct 30 20:25:34.215365 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215224 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\") pod \"fcb08303-e74d-4246-8124-5c716d720da5\" (UID: \"fcb08303-e74d-4246-8124-5c716d720da5\") " Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215489 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215633 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215650 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d25c5e7-7376-49ac-bf09-25e92359d510-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215665 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215679 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215695 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215708 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215721 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb08303-e74d-4246-8124-5c716d720da5-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216051 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.215735 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.216569 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.216058 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.216959 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.216929 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.216959 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.216935 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.217368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.217331 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.217368 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.217342 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.217545 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.217395 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.218867 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.218841 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.219146 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.219119 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.219657 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.219567 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.219657 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.219629 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.220113 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.219728 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.220347 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220321 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "kube-api-access-gzq7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.220442 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220370 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.220525 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220503 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "kube-api-access-sdmdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.220589 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220566 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.220655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220584 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk" pod "fcb08303-e74d-4246-8124-5c716d720da5" (UID: "fcb08303-e74d-4246-8124-5c716d720da5"). InnerVolumeSpecName "kube-api-access-kr8sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.220720 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220692 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy" pod "9d25c5e7-7376-49ac-bf09-25e92359d510" (UID: "9d25c5e7-7376-49ac-bf09-25e92359d510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.220922 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.220903 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.316844 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.316793 4990 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\") pod \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\" (UID: \"8413d9c9-a94c-4b84-ad92-eee68b5beac4\") " Oct 30 20:25:34.317127 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317075 4990 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs" pod "8413d9c9-a94c-4b84-ad92-eee68b5beac4" (UID: "8413d9c9-a94c-4b84-ad92-eee68b5beac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.317202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317184 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317209 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317223 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317240 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317237 4990 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb08303-e74d-4246-8124-5c716d720da5-istio-podinfo\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317250 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317263 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmdn\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-kube-api-access-sdmdn\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317277 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317290 4990 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istiod-ca-cert\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317303 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317316 4990 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-data\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317329 4990 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-istio-envoy\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317345 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317341 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8413d9c9-a94c-4b84-ad92-eee68b5beac4-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317354 4990 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-credential-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317366 4990 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-istio-token\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317380 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzq7c\" (UniqueName: \"kubernetes.io/projected/9d25c5e7-7376-49ac-bf09-25e92359d510-kube-api-access-gzq7c\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317393 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb08303-e74d-4246-8124-5c716d720da5-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317406 4990 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d25c5e7-7376-49ac-bf09-25e92359d510-workload-socket\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.317595 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.317419 4990 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8sk\" (UniqueName: \"kubernetes.io/projected/fcb08303-e74d-4246-8124-5c716d720da5-kube-api-access-kr8sk\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.418766 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:34.418699 4990 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8413d9c9-a94c-4b84-ad92-eee68b5beac4-workload-certs\") on node \"ip-10-0-135-68.ec2.internal\" DevicePath \"\"" Oct 30 20:25:35.095217 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:25:35.095182 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:25:35.188970 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.188881 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt" Oct 30 20:25:35.189453 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.189111 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5" Oct 30 20:25:35.189453 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.189289 4990 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm" Oct 30 20:25:35.251021 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.250990 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:25:35.262957 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.262930 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-czfvt"] Oct 30 20:25:35.288223 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.288200 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:25:35.291166 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.291142 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8l6gm"] Oct 30 20:25:35.327495 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.327471 4990 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:25:35.331186 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:35.331166 4990 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-vv8z5"] Oct 30 20:25:36.398713 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:36.398679 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" path="/var/lib/kubelet/pods/8413d9c9-a94c-4b84-ad92-eee68b5beac4/volumes" Oct 30 20:25:36.399285 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:36.399267 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" path="/var/lib/kubelet/pods/9d25c5e7-7376-49ac-bf09-25e92359d510/volumes" Oct 30 20:25:36.399700 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:36.399687 4990 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb08303-e74d-4246-8124-5c716d720da5" path="/var/lib/kubelet/pods/fcb08303-e74d-4246-8124-5c716d720da5/volumes" Oct 30 20:25:59.035541 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:59.035506 4990 scope.go:122] "RemoveContainer" containerID="c595a274b8b270407bf2a10b7cbdb0f9b568f43d59ce9277a708854b5388bce5" Oct 30 20:25:59.044653 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:59.044633 4990 scope.go:122] "RemoveContainer" containerID="7550ad647610d548a80f62fea9176bceb0271f8ab3167870642166d983485119" Oct 30 20:25:59.052669 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:25:59.052647 4990 scope.go:122] "RemoveContainer" containerID="8873c9a7f3b46720983d5143633ddef5be5b51840989130e9b12988d4af7202a" Oct 30 20:26:05.094881 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:26:05.094798 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:26:06.502450 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:06.502406 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:07.197066 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:07.197025 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56dcb668cc-fh7dx_cda75769-7027-40c0-8288-b1dfd4ddfe8a/authorino/0.log" Oct 30 20:26:07.444928 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:07.444897 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xrlsb_e7a72d51-4726-4114-96de-9ec15e1ab570/manager/0.log" Oct 30 20:26:07.568210 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:07.568177 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jqxt6_e3f626ef-bed4-4974-9ba4-7778435cee18/kuadrant-console-plugin/0.log" Oct 30 20:26:07.721403 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:07.721378 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-qpmtp_ad2b1fe0-8a6f-402b-a834-a6410a77fc72/manager/0.log" Oct 30 20:26:09.219211 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.219184 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/alertmanager/0.log" Oct 30 20:26:09.340795 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.340765 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/config-reloader/0.log" Oct 30 20:26:09.460104 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.460063 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/kube-rbac-proxy-web/0.log" Oct 30 20:26:09.580861 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.580839 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/kube-rbac-proxy/0.log" Oct 30 20:26:09.702799 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.702776 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/kube-rbac-proxy-metric/0.log" Oct 30 20:26:09.824449 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.824424 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/prom-label-proxy/0.log" Oct 30 20:26:09.979822 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:09.979761 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47f374c3-876b-413d-8b54-daea5cc60676/init-config-reloader/0.log" Oct 30 20:26:11.658894 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:11.658855 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5m5f_53bc3361-f8d3-49de-8490-baf98084ac36/node-exporter/0.log" Oct 30 20:26:11.783500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:11.783471 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5m5f_53bc3361-f8d3-49de-8490-baf98084ac36/kube-rbac-proxy/0.log" Oct 30 20:26:11.904909 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:11.904884 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z5m5f_53bc3361-f8d3-49de-8490-baf98084ac36/init-textfile/0.log" Oct 30 20:26:13.658587 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:13.658555 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d9f684f-ncjrn_6a6b7bd2-39e2-484b-9e37-ac5bad17363a/telemeter-client/0.log" Oct 30 20:26:13.777500 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:13.777474 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d9f684f-ncjrn_6a6b7bd2-39e2-484b-9e37-ac5bad17363a/reload/0.log" Oct 30 20:26:13.897319 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:13.897290 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-658d9f684f-ncjrn_6a6b7bd2-39e2-484b-9e37-ac5bad17363a/kube-rbac-proxy/0.log" Oct 30 20:26:15.622064 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:15.622014 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:18.215563 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215528 4990 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg"] Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215897 4990 cpu_manager.go:414] "containerMap: removing container" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215908 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215920 4990 cpu_manager.go:414] "containerMap: removing container" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215925 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215937 4990 cpu_manager.go:414] "containerMap: removing container" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" Oct 30 20:26:18.215955 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215942 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215962 4990 cpu_manager.go:414] "containerMap: removing container" podUID="70863ead-84b0-45ad-b88f-9d5019a90da1" containerName="maas-api" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.215967 4990 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70863ead-84b0-45ad-b88f-9d5019a90da1" containerName="maas-api" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.216017 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="70863ead-84b0-45ad-b88f-9d5019a90da1" containerName="maas-api" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.216028 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb08303-e74d-4246-8124-5c716d720da5" containerName="istio-proxy" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.216034 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d25c5e7-7376-49ac-bf09-25e92359d510" containerName="istio-proxy" Oct 30 20:26:18.216207 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.216041 4990 memory_manager.go:365] "RemoveStaleState removing state" podUID="8413d9c9-a94c-4b84-ad92-eee68b5beac4" containerName="istio-proxy" Oct 30 20:26:18.219194 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.219177 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.238079 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.238053 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"openshift-service-ca.crt\"" Oct 30 20:26:18.239887 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.239862 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg"] Oct 30 20:26:18.246757 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.246732 4990 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"kube-root-ca.crt\"" Oct 30 20:26:18.247036 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.247017 4990 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjhm6\"/\"default-dockercfg-st55s\"" Oct 30 20:26:18.361025 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.360998 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-proc\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.361152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.361029 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshgx\" (UniqueName: \"kubernetes.io/projected/6059979b-6e69-49c9-9171-95caffba1195-kube-api-access-rshgx\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.361152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.361050 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-lib-modules\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.361152 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.361067 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-podres\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.361274 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.361211 4990 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-sys\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.461899 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.461877 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-sys\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462011 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.461940 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-proc\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462011 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.461959 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshgx\" (UniqueName: \"kubernetes.io/projected/6059979b-6e69-49c9-9171-95caffba1195-kube-api-access-rshgx\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462011 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.461980 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-lib-modules\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462011 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.461996 4990 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-podres\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.462000 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-sys\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.462012 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-proc\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.462067 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-podres\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.462192 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.462079 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6059979b-6e69-49c9-9171-95caffba1195-lib-modules\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.481170 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.481115 4990 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshgx\" (UniqueName: \"kubernetes.io/projected/6059979b-6e69-49c9-9171-95caffba1195-kube-api-access-rshgx\") pod \"perf-node-gather-daemonset-jm5qg\" (UID: \"6059979b-6e69-49c9-9171-95caffba1195\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.531717 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.531693 4990 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:18.830655 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:18.830582 4990 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg"] Oct 30 20:26:18.833568 ip-10-0-135-68 kubenswrapper[4990]: W1030 20:26:18.833536 4990 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6059979b_6e69_49c9_9171_95caffba1195.slice/crio-a3bced5ae457b8bd4766b30aa45fbaad4c6a684c3bd1d6fac9a3b16843d52eac WatchSource:0}: Error finding container a3bced5ae457b8bd4766b30aa45fbaad4c6a684c3bd1d6fac9a3b16843d52eac: Status 404 returned error can't find the container with id a3bced5ae457b8bd4766b30aa45fbaad4c6a684c3bd1d6fac9a3b16843d52eac Oct 30 20:26:19.398070 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:19.398036 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" event={"ID":"6059979b-6e69-49c9-9171-95caffba1195","Type":"ContainerStarted","Data":"4af1cf47feb30f8b8a2cd8d337156cafc4feb81397dc949a36033fdeab940387"} Oct 30 20:26:19.398509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:19.398101 4990 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" event={"ID":"6059979b-6e69-49c9-9171-95caffba1195","Type":"ContainerStarted","Data":"a3bced5ae457b8bd4766b30aa45fbaad4c6a684c3bd1d6fac9a3b16843d52eac"} Oct 30 20:26:19.398509 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:19.398147 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:19.437474 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:19.437415 4990 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" podStartSLOduration=1.437403461 podStartE2EDuration="1.437403461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:19.43576099 +0000 UTC m=+4887.147286355" watchObservedRunningTime="2026-10-30 20:26:19.437403461 +0000 UTC m=+4887.148928827" Oct 30 20:26:21.680521 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:21.680488 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lxr6q_9712bd0c-c603-4d44-a99b-32bd83dc414c/dns/0.log" Oct 30 20:26:21.796581 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:21.796551 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lxr6q_9712bd0c-c603-4d44-a99b-32bd83dc414c/kube-rbac-proxy/0.log" Oct 30 20:26:22.288569 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:22.288537 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4kdn_e5b7c1f1-3b2d-4b73-8870-f1cffe60206e/dns-node-resolver/0.log" Oct 30 20:26:23.897065 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:23.897032 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhtch_8d9d8a4b-34d0-47ba-abc1-8e71a55d600c/node-ca/0.log" Oct 30 20:26:25.414352 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:25.414324 4990 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-jm5qg" Oct 30 20:26:28.117972 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:28.117935 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:31.585202 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:31.585172 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-dbd6g_83a243fd-8ba3-4e72-9760-6583b0458d47/kube-auth-proxy/0.log" Oct 30 20:26:34.997435 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:34.997398 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxcl2_4e563a40-316b-454d-8e73-fa7d8fd091b3/serve-healthcheck-canary/0.log" Oct 30 20:26:35.094710 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:26:35.094675 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:26:36.774629 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:36.774601 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dw5vv_ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/kube-rbac-proxy/0.log" Oct 30 20:26:36.886304 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:36.886278 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dw5vv_ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/exporter/0.log" Oct 30 20:26:36.999183 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:36.999156 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dw5vv_ae25ecb1-8d46-432e-9ef8-dcd2dd38bcab/extractor/0.log" Oct 30 20:26:42.020329 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:42.020269 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:42.818393 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:42.818355 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:43.166395 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:43.166325 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/manager/0.log" Oct 30 20:26:43.350689 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:43.350662 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ldgdt_7bd08d52-d04c-4956-b88f-c03db1cf376c/copy-manifests/0.log" Oct 30 20:26:45.263305 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:45.263273 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c54b56ff8-4bj59_eafed21c-629e-4efd-a312-aabcb81f5cd4/manager/0.log" Oct 30 20:26:56.831718 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:56.831689 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/kube-multus-additional-cni-plugins/0.log" Oct 30 20:26:56.952628 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:56.952602 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/egress-router-binary-copy/0.log" Oct 30 20:26:57.073824 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:57.073791 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/cni-plugins/0.log" Oct 30 20:26:57.193843 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:57.193758 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/bond-cni-plugin/0.log" Oct 30 20:26:57.315810 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:57.315770 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/routeoverride-cni/0.log" Oct 30 20:26:57.442184 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:57.442150 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/whereabouts-cni-bincopy/0.log" Oct 30 20:26:57.562807 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:26:57.562780 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6txgd_5d5e6381-5d5c-4d33-86cb-3e528c197bcd/whereabouts-cni/0.log" Oct 30 20:27:00.027060 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:00.027032 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2579_67705ac6-bf6a-4e9e-9749-691459286cde/kube-multus/0.log" Oct 30 20:27:00.652963 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:00.652926 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9rzn_ed069dac-23ef-4ec7-88a9-e74dc4e4161b/network-metrics-daemon/0.log" Oct 30 20:27:00.751018 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:00.750989 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9rzn_ed069dac-23ef-4ec7-88a9-e74dc4e4161b/kube-rbac-proxy/0.log" Oct 30 20:27:03.872742 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:03.872709 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-controller/0.log" Oct 30 20:27:03.992225 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:03.992185 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/1.log" Oct 30 20:27:03.992413 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:03.992295 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovn-acl-logging/0.log" Oct 30 20:27:04.085683 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.085655 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/kube-rbac-proxy-node/0.log" Oct 30 20:27:04.208390 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.208326 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/kube-rbac-proxy-ovn-metrics/0.log" Oct 30 20:27:04.330341 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.330305 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/northd/0.log" Oct 30 20:27:04.452421 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.452392 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/nbdb/0.log" Oct 30 20:27:04.574881 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.574854 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/sbdb/0.log" Oct 30 20:27:04.977882 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:04.977809 4990 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mt7n_4fd9e114-e938-4928-94e8-c9c3bd7ae37b/ovnkube-controller/0.log" Oct 30 20:27:05.096015 ip-10-0-135-68 kubenswrapper[4990]: E1030 20:27:05.095974 4990 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dw5vv" containerName="extractor" Oct 30 20:27:06.566946 ip-10-0-135-68 kubenswrapper[4990]: I1030 20:27:06.566912 4990 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5cp96_cd2c4560-125c-4c15-a15e-9e93a59e9a06/konnectivity-agent/0.log" limit=1024